Medium文章标题爬取适配问题：列表/网格布局抓取方案求助

问题说明 #

爬取Medium出版物时遇到两种文章布局：

• 常规列表布局：标题使用<h2>标签，现有代码可正常抓取
• 网格布局：标题使用<div>标签，现有代码无法识别

解决方案 #

修改CSS选择器，同时匹配列表布局的<h2>标签和网格布局的标题<div>标签。通过观察Medium页面结构，网格布局的标题div通常带有data-testid="storyTitle"属性或graf--title类名，用复合选择器可覆盖两种场景。

修改后的完整代码 #

import requests
from bs4 import BeautifulSoup as bs
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
import time


options = Options()
options.add_argument("--headless")
options.add_argument('--log-level=3')
options.add_experimental_option('excludeSwitches', ['enable-logging'])
driver = webdriver.Chrome(options=options)

class Publication:
    def __init__(self, link):
        self.link = link

    def get_articles(self):
        "Get the articles of the user/publication which was given as input"
        link = self.link
        driver.get(link)
        scroll_pause = 0.5
        # Get scroll height
        last_height = driver.execute_script("return document.documentElement.scrollHeight")
        run_time, max_run_time = 0, 1
        while True:
            iteration_start = time.time()
            # 修正滚动逻辑，避免过度滚动
            driver.execute_script("window.scrollTo(0, document.documentElement.scrollHeight);")

            # Wait to load page
            time.sleep(scroll_pause)

            # Calculate new scroll height and compare with last scroll height
            new_height = driver.execute_script("return document.documentElement.scrollHeight")
            scrolled = new_height != last_height
            timed_out = run_time >= max_run_time
            if scrolled:
                run_time = 0
                last_height = new_height
            elif not scrolled and not timed_out:
                run_time += time.time() - iteration_start
            elif not scrolled and timed_out:
                break
        # 同时匹配两种布局的标题元素，过滤空文本
        elements = driver.find_elements(By.CSS_SELECTOR, "h2, div[data-testid='storyTitle'], .graf--title")
        for x in elements:
            title_text = x.text.strip()
            if title_text:
                print(title_text)

补充说明 #

• 复合选择器覆盖了Medium两种布局下的标题元素，适配性更强
• 增加空文本过滤，避免输出无效内容
• 修正滚动脚本，原1000*滚动逻辑会导致过度滚动，调整为滚动到当前页面底部

内容的提问来源于stack exchange，提问作者Karthik Bhandary