在R中用tryCatch处理URL重定向获取最终有效链接

我有一个数据框，其中URLs列包含23000条网站重定向链接，需要获取这些链接的最终跳转URL并存储到新列中。但部分原始URL已失效，请求时会触发错误，因此需要用tryCatch处理异常。

示例URL数据 #

sample_urls <- c("https://icoholder.com/en/v2/ico/ico-redirect/4321?to=https%3A//sirinlabs.com%3Futm_source%3Dicoholder", 
"https://icoholder.com/en/v2/ico/ico-redirect/1036136?to=https%3A//dash2trade.com%3Futm_source%3Dicoholder", 
"https://icoholder.com/en/v2/ico/ico-redirect/1035284?to=https%3A//impt.io%3Futm_source%3Dicoholder", 
"https://icoholder.com/en/v2/ico/ico-redirect/1030235?to=https%3A//calvaria.io%3Futm_source%3Dicoholder", 
"https://icoholder.com/en/v2/ico/ico-redirect/1011041?to=https%3A//artyfact.art%3Futm_source%3Dicoholder", 
"https://icoholder.com/en/v2/ico/ico-redirect/1031430?to=https%3A//www.projectnexus.app%3Futm_source%3Dicoholder", 
"https://icoholder.com/en/v2/ico/ico-redirect/1005962?to=https%3A//seedon.io%3Futm_source%3Dicoholder", 
"https://icoholder.com/en/v2/ico/ico-redirect/1033498?to=https%3A//vicuna.network%3Futm_source%3Dicoholder", 
"https://icoholder.com/en/v2/ico/ico-redirect/1036409?to=https%3A//cryptoffer.io/%3Futm_source%3Dicoholder", 
"https://icoholder.com/en/v2/ico/ico-redirect/23905?to=http%3A//www.bitcoin.org/%3Futm_source%3Dicoholder", 
"https://icoholder.com/en/v2/ico/ico-redirect/1450?to=https%3A//ethereum.org%3Futm_source%3Dicoholder", 
"https://icoholder.com/en/v2/ico/ico-redirect/17581?to=https%3A//telegram.org%3Futm_source%3Dicoholder", 
"https://icoholder.com/en/v2/ico/ico-redirect/1009688?to=https%3A//egoco.in/%3Futm_source%3Dicoholder", 
"https://icoholder.com/en/v2/ico/ico-redirect/19163?to=https%3A//lapo.io%3Futm_source%3Dicoholder", 
"https://icoholder.com/en/v2/ico/ico-redirect/20971?to=https%3A//ingotcoin.io%3Futm_source%3Dicoholder", 
"https://icoholder.com/en/v2/ico/ico-redirect/26401?to=https%3A//restotoken.org%3Futm_source%3Dicoholder",
"https://icoholder.com/en/v2/ico/ico-redirect/4321?to=https%3A//ccc"
)

# 构造示例数据框
df <- data.frame(URLs = sample_urls)

当前测试代码 #

library(httr)

df$URLs <- tryCatch(sapply(df$URLs, function(x) GET(x)$url), error = function(e) return(NULL))

问题分析与代码修改 #

当前代码的问题在于tryCatch包裹了整个sapply操作，只要有一个URL触发错误，整个结果就会返回NULL，覆盖原数据列。正确的做法是将tryCatch放到单个URL的处理逻辑内部，确保单个URL出错不影响整体批量处理。

修改后的代码如下：

library(httr)

# 定义处理单个URL的函数，包含异常捕获
get_final_url <- function(url) {
  tryCatch({
    # 发送GET请求，默认跟随所有重定向
    resp <- GET(url)
    # 仅当请求成功时返回最终URL，否则返回NA
    if (http_status(resp)$category == "Success") {
      return(resp$url)
    } else {
      return(NA_character_)
    }
  }, error = function(e) {
    # 捕获所有请求错误（如无效URL、连接失败等），返回NA
    return(NA_character_)
  })
}

# 批量处理URL列，生成新列（不覆盖原始URL）
df$Final_URLs <- sapply(df$URLs, get_final_url)

额外提示 #

• 保留原始数据：不要覆盖原URLs列，方便后续排查失效URL的问题
• 反爬处理：针对23000条URL的批量请求，建议在函数中添加延迟，比如Sys.sleep(0.5)，避免触发目标网站的反爬机制导致IP被封
• 更稳定的批量处理：可以用purrr::map_chr替代sapply，返回类型更明确的字符向量，代码如下：

  library(purrr)
  df$Final_URLs <- map_chr(df$URLs, get_final_url)
  

• 错误记录：如果需要记录错误详情，可以在error函数中添加日志输出，或者将错误信息存储到单独的列中：

  get_final_url_with_error <- function(url) {
    tryCatch({
      resp <- GET(url)
      if (http_status(resp)$category == "Success") {
        list(final_url = resp$url, error_msg = NA_character_)
      } else {
        list(final_url = NA_character_, error_msg = http_status(resp)$message)
      }
    }, error = function(e) {
      list(final_url = NA_character_, error_msg = e$message)
    })
  }
  
  # 转换为数据框合并到原数据
  error_details <- do.call(rbind, sapply(df$URLs, get_final_url_with_error))
  df <- cbind(df, error_details)
  

内容的提问来源于stack exchange，提问作者Soph2010