Scrapy增量爬取会议纪要及Pipeline文件命名技术咨询

一、解决URL传递给Pipeline生成命名文件的问题 #

不需要改用CrawlSpider，用常规Spider就能搞定，核心是在Item中保留filelink字段，直接在Pipeline中调用该字段处理文件名：

1. 定义Item类：确保包含内容和链接字段

# items.py
import scrapy

class MeetingItem(scrapy.Item):
    content = scrapy.Field()
    filelink = scrapy.Field()  # 留存链接用于生成文件名

2. 第二个爬虫的解析逻辑：从CSV读链接，爬取内容并将URL存入Item

# 爬取会议内容的爬虫
import scrapy
from your_project.items import MeetingItem
import csv
import os

class MeetingContentSpider(scrapy.Spider):
    name = 'meeting_content'
    allowed_domains = ['your-target-domain.com']

    def start_requests(self):
        # 读取存储链接的CSV文件
        with open('meeting_links.csv', 'r', encoding='utf-8') as f:
            reader = csv.DictReader(f)
            for row in reader:
                yield scrapy.Request(url=row['filelink'], callback=self.parse_content)

    def parse_content(self, response):
        item = MeetingItem()
        # 替换为实际的内容提取XPath/CSS选择器
        item['content'] = response.xpath('//div[@class="meeting-detail"]//text()').getall()
        item['filelink'] = response.url  # 直接从response获取URL，避免传递丢失
        yield item

3. Pipeline处理文件命名与保存：清洗URL中的特殊字符，生成合法文件名并写入内容

# pipelines.py
import os
from urllib.parse import quote

class MeetingContentPipeline:
    def process_item(self, item, spider):
        # 清洗URL：替换特殊字符，避免系统无法创建文件
        clean_filename = quote(item['filelink'], safe='') + '.txt'
        # 创建保存目录（不存在则新建）
        save_dir = 'meeting_records'
        if not os.path.exists(save_dir):
            os.makedirs(save_dir)
        # 写入内容到对应文件
        with open(os.path.join(save_dir, clean_filename), 'w', encoding='utf-8') as f:
            f.write('\n'.join(item['content']))
        return item

在settings.py中启用该Pipeline：

ITEM_PIPELINES = {
    'your_project.pipelines.MeetingContentPipeline': 300,
}

二、增量爬取新增链接的实现 #

第一个爬虫需要对比已爬链接，仅导出新增内容：

1. 存储已爬链接：用文本文件（或SQLite）记录所有已爬过的链接，方便快速去重
2. 第一个爬虫的核心逻辑：

# 爬取会议链接的爬虫
import scrapy
from your_project.items import MeetingLinkItem
import os

class MeetingLinkSpider(scrapy.Spider):
    name = 'meeting_link'
    allowed_domains = ['your-target-domain.com']
    start_urls = ['https://your-target-domain.com/meeting-list']

    def __init__(self):
        # 读取已爬链接存入集合，实现O(1)查询
        self.crawled_links = set()
        if os.path.exists('crawled_links.txt'):
            with open('crawled_links.txt', 'r', encoding='utf-8') as f:
                self.crawled_links = set(f.read().splitlines())

    def parse(self, response):
        # 提取页面上的所有会议链接
        raw_links = response.xpath('//a[@class="meeting-entry"]/@href').getall()
        for link in raw_links:
            full_link = response.urljoin(link)
            # 仅处理未爬取过的链接
            if full_link not in self.crawled_links:
                item = MeetingLinkItem()
                item['filelink'] = full_link
                yield item
                # 实时写入已爬列表，避免中断后重复爬取
                with open('crawled_links.txt', 'a', encoding='utf-8') as f:
                    f.write(full_link + '\n')
        # 处理分页（如果有）
        next_page = response.xpath('//a[@rel="next"]/@href').get()
        if next_page:
            yield scrapy.Request(url=response.urljoin(next_page), callback=self.parse)

3. Pipeline导出新增链接到CSV：

# pipelines.py 新增链接导出Pipeline
from scrapy.exporters import CsvItemExporter
import os

class MeetingLinkPipeline:
    def open_spider(self, spider):
        # 以追加模式打开CSV，避免覆盖历史数据
        self.file = open('meeting_links.csv', 'a', encoding='utf-8', newline='')
        self.exporter = CsvItemExporter(self.file)
        # 如果文件为空，写入表头
        if os.path.getsize('meeting_links.csv') == 0:
            self.exporter.start_exporting()
            self.exporter.export_item({'filelink': 'filelink'})

    def process_item(self, item, spider):
        self.exporter.export_item(item)
        return item

    def close_spider(self, spider):
        self.exporter.finish_exporting()
        self.file.close()

调整settings.py中的Pipeline优先级，确保链接导出先执行：

ITEM_PIPELINES = {
    'your_project.pipelines.MeetingLinkPipeline': 200,
    'your_project.pipelines.MeetingContentPipeline': 300,
}

三、注意事项 #

• URL清洗必须做：URL中的/、?、=等特殊字符会导致系统无法创建文件，用urllib.parse.quote或手动替换为合法字符均可
• 增量爬取的可靠性：用集合存储已爬链接，查询速度快；实时写入已爬列表，避免爬虫中断后重复爬取
• 数据量大时可改用SQLite替代文本文件存储已爬链接，提升查询和维护效率

内容的提问来源于stack exchange，提问作者areeekay