使用rvest抓取bioguide网站图片alt属性失败，求解决方案

问题描述 #

尝试数周抓取https://bioguide.congress.gov/search网站的所有图片，当前目标是提取img标签的alt属性内容，但运行R代码后返回character(0)。

目标HTML片段 #

<div class="l-grid__item l-grid__item--3/12 l-grid__item--12/12@mobile--sm l-grid__item--4/12@desktop l-grid__item--6/12@tablet"><div tabindex="0" class="c-card u-flex u-flex--column u-height--100% u-cursor--pointer u-bxs--dark-lg:hover c-card--@print"><div class="u-height--100% u-width--100% u-p u-flex u-flex--centered u-mb--auto"><div aria-hidden="true" class="u-max-width--80% u-max-height--250px"><img alt="/photo/66c88d1d7401a93215e0b225.jpg" class="u-max-height--250px u-height--auto u-width--auto u-block" src="/photo/66c88d1d7401a93215e0b225.jpg"></div></div><div class="u-flex u-flex--column u-flex--no-shrink u-p u-bg--off-white u-fw--bold u-color--primary u-text--center u-bt--light-gray"><div class="u-cursor--pointer u-mb--xs">AANDAHL, Fred George</div><div class="u-fz--sm u-fw--semibold">1897 – 1966</div></div></div></div>

原R代码 #

library(httr)
library(rvest)

# Fetch the HTML content with a custom User-Agent
response <- GET("https://bioguide.congress.gov/search", 
                user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36"))

# Parse the content
page <- read_html(content(response, as = "text", encoding = "UTF-8"))

# Navigate to the div with class starting with 'l-grid__item' and extract img alt attributes
img_alt_values <- page %>%
  html_nodes(xpath = "//div[starts-with(@class, 'l-grid__item')]") %>%
  html_nodes(xpath = ".//img") %>%
  html_attr("alt")

问题原因 #

该网站采用动态渲染机制，直接用GET请求获取的是初始空页面HTML，页面上的议员卡片内容是通过前端JavaScript异步加载的，所以你的XPath无法定位到目标节点。

解决方法 #

方法1：调用官方API（推荐，高效可靠） #

抓包发现网站通过API接口获取议员数据，直接请求API可拿到结构化的图片路径，无需解析HTML：

library(httr)
library(jsonlite)

# 发送POST请求获取议员数据
response <- POST(
  url = "https://bioguide.congress.gov/search/bio",
  add_headers(
    "User-Agent" = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36",
    "Content-Type" = "application/json"
  ),
  body = toJSON(list(
    query = "",
    offset = 0,
    limit = 100, # 每次请求100条，可根据需求调整
    sort = "lastname asc"
  ), auto_unbox = TRUE)
)

# 解析JSON响应
data <- fromJSON(content(response, as = "text"))

# 提取图片路径（alt与src属性值一致）
img_paths <- data$results$photo

# 查看结果
print(img_paths)

# 循环获取所有数据示例
all_img_paths <- c()
offset <- 0
while(TRUE) {
  response <- POST(
    url = "https://bioguide.congress.gov/search/bio",
    add_headers(
      "User-Agent" = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36",
      "Content-Type" = "application/json"
    ),
    body = toJSON(list(
      query = "",
      offset = offset,
      limit = 100,
      sort = "lastname asc"
    ), auto_unbox = TRUE)
  )
  data <- fromJSON(content(response, as = "text"))
  if(length(data$results) == 0) break
  all_img_paths <- c(all_img_paths, data$results$photo)
  offset <- offset + 100
  Sys.sleep(1) # 避免请求过于频繁
}

# 所有图片路径
print(all_img_paths)

方法2：用RSelenium模拟浏览器渲染 #

通过模拟真实浏览器加载页面，获取渲染后的完整HTML：

library(RSelenium)
library(rvest)

# 启动Chrome驱动（需提前安装对应版本的ChromeDriver）
driver <- rsDriver(browser = "chrome", chromever = "92.0.4515.159")
remDr <- driver$client

# 访问目标页面
remDr$navigate("https://bioguide.congress.gov/search")

# 等待页面加载完成（根据网络情况调整等待时间）
Sys.sleep(5)

# 获取渲染后的页面源码
page_source <- remDr$getPageSource()[[1]]
page <- read_html(page_source)

# 提取img的alt属性
img_alt_values <- page %>%
  html_nodes(xpath = "//div[starts-with(@class, 'l-grid__item')]//img") %>%
  html_attr("alt")

# 查看结果
print(img_alt_values)

# 关闭浏览器和驱动
remDr$close()
driver$server$stop()

内容的提问来源于stack exchange，提问作者sfyn