Scrapy爬取aa-dc.org站点时Xpath错误导致数据重复的解决方法

问题场景 #

我正尝试从站点https://aa-dc.org/meetings?tsml-day=any&tsml-type=IPM爬取会议数据，编写了如下初始爬虫脚本：

import scrapy


class WaiascrapSpider(scrapy.Spider):
    name = 'waiascrap'
    allowed_domains = ['clsaa-dc.org']
    start_urls = ['https://aa-dc.org/meetings?tsml-day=any&tsml-type=IPM']

    def parse(self, response):
        rows = response.xpath("//tr")
        for row in rows:
            day = rows.xpath("(//tr/td[@class='time']/span)[1]/text()").get()
            time = rows.xpath("//tr/td[@class='time']/span/time/text()").get()

            yield{
                'day': day,
                'time': time,
            }

运行后获取到的所有数据完全重复，和for循环没有正常遍历每一行的表现一致，运行日志如下：

PS C:\Users\gasgu\PycharmProjects\ScrapingProject\projects\waia> scrapy crawl waiascrap
2021-08-20 15:25:11 [scrapy.utils.log] INFO: Scrapy 2.5.0 started (bot: waia)
2021-08-20 15:25:11 [scrapy.utils.log] INFO: Versions: lxml 4.6.3.0, libxml2 2.9.5, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 21.7.0, Python 3.9.6 (tags/v3.9.6:db3ff76, Jun 28 2021, 15:26:21) [MSC v.1929 64 bit (AMD64)], pyOpenSSL 20.0.1 (OpenSSL 1.1.1k 25 Mar 2021), cryptography 3.4.7, Platform Windows-10-10.0.19042-SP0
...（中间日志省略）
2021-08-20 15:25:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://aa-dc.org/meetings?tsml-day=any&tsml-type=IPM> (referer: None)
2021-08-20 15:25:16 [scrapy.core.scraper] DEBUG: Scraped from <200 https://aa-dc.org/meetings?tsml-day=any&tsml-type=IPM> {'day': 'Sunday', 'time': '6:45 am'}
2021-08-20 15:25:19 [scrapy.core.scraper] DEBUG: Scraped from <200 https://aa-dc.org/meetings?tsml-day=any&tsml-type=IPM> {'day': 'Sunday', 'time': '6:45 am'}
...（所有返回结果完全重复）

问题原因 #

代码存在两处Xpath语法错误：

1. 循环内调用对象错误：遍历每行row时，误用了存储所有行的集合变量rows调用xpath方法，导致每次匹配都是全局范围。
2. Xpath路径作用域错误：Xpath前缀//代表从整个HTML文档根节点开始搜索，哪怕在单个行节点下执行，也会返回全局匹配的第一个结果，不会限定在当前行内。

修正后代码 #

将循环内的xpath调用对象改为当前行row，同时给Xpath路径加上前缀.，代表从当前节点开始向下搜索，即可正常遍历每行获取对应数据：

import scrapy


class WaiascrapSpider(scrapy.Spider):
    name = 'waiascrap'
    allowed_domains = ['clsaa-dc.org']
    start_urls = ['https://aa-dc.org/meetings?tsml-day=any&tsml-type=IPM']

    def parse(self, response):
        rows = response.xpath("//tr")
        for row in rows:
            day = row.xpath(".//td[@class='time']/span/text()").get()
            time = row.xpath(".//td[@class='time']/span/time/text()").get()
            yield {
                'day': day,
                'time': time,
                }

内容的提问来源于stack exchange，提问作者Baldie47