使用R语言多页爬取WhoSampled网站数据的问题求助

爬取需求 #

从https://www.whosampled.com/Daft-Punk/sampled/?role=1获取Daft Punk所有曲目链接，再从每首曲目详情页提取采样该曲目的艺人、采样曲目、年份、类型信息（例如首曲《Harder, Better, Faster, Stronger》的详情页为https://www.whosampled.com/Daft-Punk/Harder,-Better,-Faster,-Stronger/sampled/）。

编写的代码 #

library(tidyverse)
library(rvest)
library(httr)

#setwd for useragents, i will upload on web when i will finish
setwd("C:/Users/c_ans/Desktop/bologna lezioni/comunication of statistics")
alist<-read.table("useragents.txt", sep = "\n")
#create a list of links with all the possible songs
#page <- read_html("https://www.whosampled.com/Daft-Punk/sampled/?role=1", user_agent=sample(ualist))
#create a list of links with all the possible songs
page <- read_html("https://www.whosampled.com/Daft-Punk/sampled/?role=1")
tracks_links <- page %>% html_nodes(".trackCover") %>%
    html_attr("href") %>% paste0("https://www.whosampled.com", .,",sampled/") #href is used to extract the url for each song

#first method
#loop for each link
for (i in 1:length(tracks_links)){
  Sys.sleep(5)
    page <- read_html(tracks_links[i], user_agent=sample(ualist))
    pages <- page %>% 
  html_elements(".page a") %>% 
  html_text2() %>% 
  last()
  #extract the data from each page
    paste0(tracks_links[i], "?cp=", 1:pages) %>%
     #map(~read_html(.,user_agent=sample(ualist))) #i think the problem is here, i would like to apply an user agent each time the loop is executed
        map(read_html) %>% 
        #map_dfr(~ html_elements(.x, ".table.tdata tbody tr") %>% 
                  map_dfr(~ html_elements(.x, ".table.tdata tbody tr") %>% 
            map_dfr(~ tibble(
              title = html_element(.x, ".trackName.playIcon") %>% 
                html_text2(),
              artist = html_element(.x, ".tdata__td3") %>% 
                html_text2(),
              year = html_element(.x, ".tdata__td3:nth-child(4)") %>% 
                html_text2(),
              genre = html_element(.x, ".tdata__badge") %>% 
                html_text2()
                    )))
}

遇到的问题 #

• 运行循环时出现Error in open.connection(x, "rb") : HTTP error 403.，推测是网站反爬机制导致，使用user-agent仍无法解决；
• 循环无法执行完成，单URL测试内部逻辑有效，但不确定paste0(tracks_links[i], "?cp=", 1:pages) %>% map(~read_html(.,user_agent=sample(ualist)))代码正确性；
• 首次使用tibble存储数据，不确定当前存储逻辑是否正确。

解决方案 #

1. 解决403反爬问题 #

403错误核心是网站识别出爬虫行为，需从多维度伪装请求：

• 修正User-Agent调用错误：代码中ualist未定义，实际读取的是alist，需改为sample(alist$V1, 1)（read.table读入的是data.frame，要取第一列数据）；
• 补充请求头：模拟真实浏览器添加Accept、Accept-Language、Referer等字段，示例：

  req <- GET(url, user_agent(sample(alist$V1,1)), 
             add_headers(
               "Accept" = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
               "Accept-Language" = "en-US,en;q=0.5",
               "Referer" = "https://www.whosampled.com/"
             ))
  page <- read_html(content(req, "text"))
  

• 随机延长休眠时间：替换固定Sys.sleep(5)为Sys.sleep(sample(3:8,1))，避免固定请求间隔被检测；
• 分批爬取：若曲目数量多，可每爬取5个链接后暂停10秒，降低请求频率。

2. 修正分页与循环逻辑 #

• 修复曲目链接拼接错误：原代码中paste0("https://www.whosampled.com", .,",sampled/")的逗号是错误的，应改为paste0("https://www.whosampled.com", ., "/sampled/")，否则生成无效链接；
• 完善分页判断：当页面只有1页时，pages会为空，需添加判断：

  pages <- page %>% html_elements(".page a") %>% html_text2()
  pages <- ifelse(length(pages) == 0, 1, as.integer(last(pages)))
  

• map中正确传递User-Agent：在map的匿名函数中每次请求都随机选择User-Agent，同时修正HTML选择器（原选择器.tdata__td3对应列错误，艺人实际在第二列）：

  paste0(tracks_links[i], "?cp=", 1:pages) %>%
    map(function(url) {
      Sys.sleep(sample(2:5,1))
      req <- GET(url, user_agent(sample(alist$V1,1)), 
                 add_headers(
                   "Accept" = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
                   "Accept-Language" = "en-US,en;q=0.5",
                   "Referer" = tracks_links[i]
                 ))
      read_html(content(req, "text"))
    }) %>%
    map_dfr(function(page) {
      page %>% html_elements(".table.tdata tbody tr") %>%
        map_dfr(function(row) {
          tibble(
            daft_punk_track = basename(dirname(tracks_links[i])), # 关联对应的Daft Punk曲目
            sampled_title = html_element(row, ".trackName.playIcon") %>% html_text2(),
            sampled_artist = html_element(row, ".tdata__td2") %>% html_text2(),
            sampled_year = html_element(row, ".tdata__td4") %>% html_text2(),
            sampled_genre = html_element(row, ".tdata__badge") %>% html_text2()
          )
        })
    })
  

3. 完善tibble存储逻辑 #

• 初始化全局数据框：循环前定义final_data <- tibble()，每次循环后用final_data <- bind_rows(final_data, current_data)合并结果；
• 添加关联字段：加入daft_punk_track字段，明确每条数据对应的Daft Punk曲目；
• 处理空值：用replace_na填充缺失的年份或类型，避免数据断裂。

内容的提问来源于stack exchange，提问作者GiulioSurya