如何修正R爬虫获取医学词典网站<h2>内容及术语表失败问题

问题根源 #

原代码错误选择了页面头部标题元素，且未意识到该网站术语按字母分类展示在二级页面，首页仅提供字母导航入口，导致抓取无效内容。

修正后的完整代码 #

library(rvest)
library(dplyr)

scrape_medical_dict <- function(base_url) {
  # 1. 爬取首页的字母导航链接
  homepage <- read_html(base_url)
  letter_links <- homepage %>%
    html_elements(".alphabetLinks a") %>%
    html_attr("href") %>%
    paste0(base_url, .) # 拼接完整URL
  
  # 2. 遍历每个字母页面，爬取术语和释义
  all_terms <- data.frame()
  for(link in letter_links) {
    page <- read_html(link)
    
    # 获取术语（每个术语在.cardHeader下的h3元素）
    terms <- page %>%
      html_elements(".cardHeader h3") %>%
      html_text2() %>%
      trimws()
    
    # 获取对应释义（每个术语的释义在.definitionParagraph元素）
    definitions <- page %>%
      html_elements(".definitionParagraph") %>%
      html_text2() %>%
      trimws()
    
    # 合并成数据框，确保术语和释义数量匹配
    page_df <- data.frame(
      term = terms,
      defn = definitions,
      stringsAsFactors = FALSE
    )
    
    all_terms <- bind_rows(all_terms, page_df)
  }
  
  # 添加序号列
  all_terms <- all_terms %>%
    mutate(id = row_number()) %>%
    select(id, term, defn)
  
  return(all_terms)
}

# 调用函数
base_url <- "https://medicaldictionary.lib.umich.edu/"
result_table <- scrape_medical_dict(base_url)

# 查看前5行结果
head(result_table, 5)

关键修正说明 #

• 导航逻辑修正：先抓取首页字母导航链接，再逐个访问对应术语列表页面，这是获取全量术语的核心前提。
• CSS选择器修正：
  • 术语选择.cardHeader h3，精准定位每个卡片的医学术语标题。
  • 释义选择.definitionParagraph，直接抓取对应术语的释义文本。
• 数据匹配：每个字母页面内术语与释义数量一一对应，直接按顺序合并即可，无需补全NA。
• 序号添加：通过row_number()生成与示例格式一致的序号列。

输出示例 #

id             term                                                                                                               defn
1  1  24-hour coverage            When many health plans or programs are combined into one plan that can cover someone for the whole day.
2  2         abatement         Remove or cover a dangerous material, such as lead or mercury, from water, paint, or air, or other places.
3  3 abdomen, abdominal                                                                                          Your belly or tummy area.
4  4           ability                                                                                              To be able to or can do.
5  5          ablation To remove or destroy a body part or tissue. This can be done in many ways, including surgery, radiation, or drugs.

内容的提问来源于stack exchange，提问作者stats_noob