使用Scrapy爬取网页无输出问题排查求助

问题描述 #

我想爬取目标页面中所有建筑的名称，以及每个建筑链接里的更多数据，使用Scrapy的CrawlSpider实现，但运行脚本后没有任何输出。附上代码和终端日志，请求排查问题。

爬取代码 #

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
#from scrapy.crawler import CrawlerProcess


class AkershusSpider(CrawlSpider):
    name = 'akershus'
    allowed_domains = ['akershuseiendom.no']
    start_urls = ['http://akershuseiendom.no/lokaler',]

    rules = (
        Rule(LinkExtractor(restrict_xpaths='//*[@class="items animate-3"]/a'), callback='parse_item', follow=True),
    )

    def parse_item(self, response):
       yield {
         'Bygget': response.xpath('//*[@id="html"]/body/div[1]/main/article/header/div[2]/h1').get()  
       }
       
#process=CrawlerProcess()
#process.crawl(AkershusSpider)
#process.start()

终端日志片段 #

2022-11-23 23:49:53 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://akershuseiendom.no/lokaler> from <GET http://akershuseiendom.no/lokaler>
2022-11-23 23:49:54 [filelock] DEBUG: Attempting to acquire lock 2380446210848 on C:\Users\103720nutr\Documents\scripts\eiendomsverdi\venv\lib\site-packages\tldextract\.suffix_cache\publicsuffix.org-tlds\de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2022-11-23 23:49:54 [filelock] DEBUG: Lock 2380446210848 acquired on C:\Users\103720nutr\Documents\scripts\eiendomsverdi\venv\lib\site-packages\tldextract\.suffix_cache\publicsuffix.org-tlds\de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2022-11-23 23:49:54 [filelock] DEBUG: Attempting to release lock 2380446210848 on C:\Users\103720nutr\Documents\scripts\eiendomsverdi\venv\lib\site-packages\tldextract\.suffix_cache\publicsuffix.org-tlds\de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2022-11-23 23:49:54 [filelock] DEBUG: Lock 2380446210848 released on C:\Users\103720nutr\Documents\scripts\eiendomsverdi\venv\lib\site-packages\tldextract\.suffix_cache\publicsuffix.org-tlds\de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2022-11-23 23:49:54 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://akershuseiendom.no/lokaler> (referer: None)
2022-11-23 23:49:54 [scrapy.core.engine] INFO: Closing spider (finished)
2022-11-23 23:49:54 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 450,
 'downloader/request_count': 2,

问题排查与修复 #

1. LinkExtractor规则失效 #

日志显示爬虫仅爬取起始页面就结束，说明未提取到任何符合规则的链接，核心问题在restrict_xpaths表达式：

• 原代码中使用的"是HTML转义字符，Scrapy无法识别，需替换为普通双引号"
• 调整XPath为更精准的结构，比如//div[@class="items animate-3"]/a，匹配目标元素

2. 起始URL的重定向优化 #

起始URL为http协议，被301重定向到https，虽Scrapy会自动处理，但建议直接将start_urls改为https://akershuseiendom.no/lokaler，减少额外请求

3. 建筑名称提取XPath简化 #

原XPath路径过长，易因页面结构变动失效，简化为更稳定的表达式，比如：

'Bygget': response.xpath('//article/header/div[2]/h1/text()').get()

4. 爬虫运行方式确认 #

• 若用Scrapy命令行（scrapy crawl akershus）启动，注释的CrawlerProcess代码无需修改
• 若直接运行脚本，需取消注释CrawlerProcess相关代码，否则无法启动爬虫

修复后示例代码 #

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from scrapy.crawler import CrawlerProcess


class AkershusSpider(CrawlSpider):
    name = 'akershus'
    allowed_domains = ['akershuseiendom.no']
    start_urls = ['https://akershuseiendom.no/lokaler',]

    rules = (
        Rule(LinkExtractor(restrict_xpaths='//div[@class="items animate-3"]/a'), callback='parse_item', follow=True),
    )

    def parse_item(self, response):
       yield {
         'Bygget': response.xpath('//article/header/div[2]/h1/text()').get()  
       }
       
process=CrawlerProcess()
process.crawl(AkershusSpider)
process.start()

内容的提问来源于stack exchange，提问作者Nunotrt