Scrapy爬虫抓取StackOverflow问题时第三列无数据求助

问题描述 #

我尝试编写Scrapy爬虫抓取StackOverflow的问题数据，但第三列（c_desc字段）无法获取到数据，以下是我的爬虫代码：

from scrapy.item import Field
from scrapy.item import Item
from scrapy.spiders import Spider
from scrapy.selector import Selector
from scrapy.loader import ItemLoader

class Question(Item):
    a_id = Field()
    b_question = Field()
    c_desc = Field()

class StackOverflowSpider(Spider):
    name = "MyFirstSpider"
    custom_settings = {
        'USER-AGENT': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36"
    }
    start_urls = ['https://stackoverflow.com/questions']

    def parse(self, response):
        sel = Selector(response)
        questions = sel.xpath('//div[@id="questions"]//div[@class="s-post-summary--content"]')
        i = 1
        for quest in questions:
            item = ItemLoader(Question(), quest)
            item.add_xpath('b_question', './/h3/a/text()')
            item.add_xpath('c_desc', './/div[@class="s-post-summary--content-excerpt"]/text()')
            item.add_value('a_id', i)
            i = i+1
            yield item.load_item()

附CSV输出情况：第三列c_desc无数据；网页HTML结构显示问题描述文本嵌套在class="s-post-summary--content-excerpt"的div内部子节点中。

解决方法 #

问题出在c_desc字段的XPath选择器和文本处理上，修改如下：

1. 导入文本处理工具 #

在代码顶部添加以下导入：

from scrapy.loader.processors import MapCompose
from w3lib.html import remove_tags

2. 修改c_desc的XPath与处理逻辑 #

将parse方法中item.add_xpath('c_desc', ...)一行替换为：

item.add_xpath('c_desc', './/div[@class="s-post-summary--content-excerpt"]//text()', MapCompose(str.strip, remove_tags))

修改后的完整代码 #

from scrapy.item import Field
from scrapy.item import Item
from scrapy.spiders import Spider
from scrapy.selector import Selector
from scrapy.loader import ItemLoader
from scrapy.loader.processors import MapCompose
from w3lib.html import remove_tags

class Question(Item):
    a_id = Field()
    b_question = Field()
    c_desc = Field()

class StackOverflowSpider(Spider):
    name = "MyFirstSpider"
    custom_settings = {
        'USER-AGENT': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36"
    }
    start_urls = ['https://stackoverflow.com/questions']

    def parse(self, response):
        sel = Selector(response)
        questions = sel.xpath('//div[@id="questions"]//div[@class="s-post-summary--content"]')
        i = 1
        for quest in questions:
            item = ItemLoader(Question(), quest)
            item.add_xpath('b_question', './/h3/a/text()')
            item.add_xpath('c_desc', './/div[@class="s-post-summary--content-excerpt"]//text()', MapCompose(str.strip, remove_tags))
            item.add_value('a_id', i)
            i = i+1
            yield item.load_item()

原理说明 #

• 使用.//div[@class="s-post-summary--content-excerpt"]//text()替代原XPath，能获取该div下所有层级的文本内容，避免因文本嵌套在子节点中而无法捕获；
• MapCompose(str.strip, remove_tags)会依次执行：去除文本前后空白字符、清理残留的HTML标签，最终得到干净的问题描述文本。

内容的提问来源于stack exchange，提问作者drone2700