R语言网页爬取遇HTTP error 404及循环逻辑问题求助

问题描述 #

爬取多个链接信息时触发Error in open.connection(x, "rb") : HTTP error 404错误，尝试转换numbers数据类型无效；使用map_df代码时，未生成目标数据框，仅在控制台打印结果。

原for循环代码 #

library(rvest)
library(tidyverse)

pageMen = read_html('https://www.bjjcompsystem.com/tournaments/1869/categories')
get_links <- pageMen %>% 
  html_nodes('.categories-grid__category a') %>% 
  html_attr('href') %>%
  paste0('https://www.bjjcompsystem.com', .) 

# extract numerical part of link
numbers = str_sub(get_links, - 7, - 1)  
numbers = as.numeric(numbers)

## create empty vector  ----------------------------
master1.tree = data.frame()

## Create for loop ---------------------------------
for (i in length(numbers)){
  url <- read_html(paste0('https://www.bjjcompsystem.com/tournaments/1869/categories/', i))
  
ageDivision <- url %>% html_nodes('.category-title__age-division') %>% html_text()

gender <- url %>% html_nodes('.category-title__age-division+ .category-title__label') %>% html_text()  

matches = data.frame('division' = ageDivision,'gender' = gender)
master1.tree <- rbind(master1.tree, data.frame(matches))
}

原map_df代码 #

map_df(get_links, function(i){
  url <- read_html(i)
  
matches <- data.frame(ageDivision <- url %>% 
  html_nodes('.category-title__age-division') %>% html_text(),
gender <- url %>% html_nodes('.category-title__age-division+ .category-title__label') %>% html_text() ) 

master1.tree <- rbind(master1.tree, matches)
})

问题排查与修正 #

1. for循环的核心问题 #

• 循环范围错误：for (i in length(numbers))只会执行一次循环，取的是numbers的长度值（比如如果有15个元素，i=15），而非遍历每个元素。正确写法是for (num in numbers)直接遍历数字，或者for (i in seq_along(numbers))通过索引取值。
• URL拼接错误：原代码用循环变量i拼接URL，但i是长度值而非实际的分类数字，导致访问不存在的页面触发404。
• 空数据框初始化不规范：直接data.frame()初始化可能导致后续rbind时类型冲突，建议指定列类型。

修正后的for循环代码

library(rvest)
library(tidyverse)

pageMen = read_html('https://www.bjjcompsystem.com/tournaments/1869/categories')
get_links <- pageMen %>% 
  html_nodes('.categories-grid__category a') %>% 
  html_attr('href') %>%
  paste0('https://www.bjjcompsystem.com', .) 

# 提取链接末尾的数字部分
numbers = str_sub(get_links, -7, -1)  
numbers = as.numeric(numbers)

# 规范初始化空数据框，指定列类型
master1.tree = data.frame(division = character(), gender = character(), stringsAsFactors = FALSE)

# 遍历每个分类数字
for (num in numbers){
  # 拼接正确的目标URL
  url <- read_html(paste0('https://www.bjjcompsystem.com/tournaments/1869/categories/', num))
  
  ageDivision <- url %>% html_nodes('.category-title__age-division') %>% html_text()
  gender <- url %>% html_nodes('.category-title__age-division+ .category-title__label') %>% html_text()  
  
  matches = data.frame(division = ageDivision, gender = gender, stringsAsFactors = FALSE)
  master1.tree <- rbind(master1.tree, matches)
}

2. map_df代码的核心问题 #

• 返回值缺失：map_df依赖函数返回的数据框进行合并，原代码未返回matches，而是试图修改全局变量，导致最终无有效数据生成。
• 数据框列名错误：创建matches时用ageDivision <- ...的赋值语法，会导致列名异常，应该用=指定列名。

修正后的map_df代码

library(rvest)
library(tidyverse)

pageMen = read_html('https://www.bjjcompsystem.com/tournaments/1869/categories')
get_links <- pageMen %>% 
  html_nodes('.categories-grid__category a') %>% 
  html_attr('href') %>%
  paste0('https://www.bjjcompsystem.com', .) 

# 利用map_df自动合并结果，函数直接返回数据框
master1.tree <- map_df(get_links, function(i){
  url <- read_html(i)
  
  ageDivision <- url %>% html_nodes('.category-title__age-division') %>% html_text()
  gender <- url %>% html_nodes('.category-title__age-division+ .category-title__label') %>% html_text()
  
  # 返回指定列名的数据框，map_df自动合并
  data.frame(division = ageDivision, gender = gender, stringsAsFactors = FALSE)
})

内容的提问来源于stack exchange，提问作者bandcar