R语言爬虫问题：如何按URL聚合提取的<p>标签文本

我是R语言新手，正在从同一招聘门户的多个URL爬取职位信息，已成功获取公司名称、职位名称、工作地点及公司官网的职位详情链接。目前爬取职位详情文本时遇到两个问题：

• 使用rvest::html_text()会抓取大量无关HTML代码，即便用自定义清洗函数处理后仍有残留；
• 指定rvest::html_nodes("p")提取目标文本后，文本按句子单独存储，无法按URL聚合（希望每个URL的全部文本可通过all_job_post_texts[[1]]这类方式调用）。

我的清洗函数 #

clean_text <- function(text) {
cleaned_text <- text %>%
str_replace_all(pattern = "\\n", replacement = " ") %>%
str_replace_all(pattern = "\\[\\^\\]", replacement = " ") %>%
str_replace_all(pattern = "\"", replacement = " ") %>%
str_replace_all(pattern = "\\s+", replacement = " ") %>%
str_trim(side = "both") %>%
substr(start = nchar(text) - 700, stop = nchar(text))

return(cleaned_text)
}

我的职位文本提取函数 #

#Extracting links to official job posts on company websites
extract_job_post_links <- function(page) {
  job_post_links <- tryCatch(
    {
      page %>%
        rvest::html_nodes("div h4 a") %>%
        rvest::html_attr("href")
    },
    error = function(e) {
      print(paste("Error:", conditionMessage(e)))
      return(NULL)  #Return NULL when an error occurs
    }
  )
  
  if (!is.null(job_post_links)) {
    #Printing the URLs as scraping them
    print(paste("Scraping URLs from:", url))
    for (link in job_post_links) {
      print(paste("  -", link))
    }
    
    #Create a list to store data from each link
    link_texts <- list()
    
    #Loop through each link and extract text
    for (link in job_post_links) {
      #Access each URL and read the HTML content
      link_page <- tryCatch(
        {
          read_html(link)
        },
        error = function(e) {
          print(paste("Error accessing URL:", link, "Error:", conditionMessage(e)))
          return(NULL)  #Return NULL when an error occurs
        }
      )
      
      if (!is.null(link_page)) {
        #Extract text from the linked page
        link_text <- link_page %>%
          rvest::html_nodes("p") %>%
          rvest::html_text()
        
        #Append the text to the list
        link_texts <- append(link_texts, link_text)
      }
    }
    
    #Return the list of link texts
    return(link_texts)
  } else {
    return(NULL)  #Return NULL when an error occurs
  }
}

解决方案 #

1. 实现按URL聚合

标签文本 #

当前代码用append把每个<p>文本单独加入列表，导致无法按URL分组。只需将单个URL下所有<p>文本合并为一个字符串，再作为列表元素添加：

修改循环内的文本提取逻辑：

if (!is.null(link_page)) {
  # 提取所有<p>文本并合并为单个字符串
  link_text <- link_page %>%
    rvest::html_nodes("p") %>%
    rvest::html_text() %>%
    paste(collapse = " ")  # 用空格拼接所有段落
  
  # 每个URL对应列表的一个元素
  link_texts[[length(link_texts) + 1]] <- link_text
}

修改后link_texts列表的每个元素对应一个URL的完整职位详情，可通过link_texts[[1]]调用第一个URL的内容。

2. 优化文本清洗，减少无关内容残留 #

原清洗函数的substr截取最后700字符可能保留无关内容，建议先过滤空文本再统一清洗：

优化后的清洗函数：

clean_text <- function(text) {
  cleaned_text <- text %>%
    # 过滤空字符串
    Filter(function(x) x != "", .) %>%
    paste(collapse = " ") %>%
    str_replace_all(pattern = "\\n|\\r", replacement = " ") %>%
    str_replace_all(pattern = "\\[\\^\\]|\"", replacement = " ") %>%
    str_replace_all(pattern = "\\s+", replacement = " ") %>%
    str_trim(side = "both")
  
  return(cleaned_text)
}

使用时直接在提取文本后调用：

link_text <- link_page %>%
  rvest::html_nodes("p") %>%
  rvest::html_text() %>%
  clean_text()

整合后的完整提取函数 #

extract_job_post_links <- function(page) {
  job_post_links <- tryCatch(
    {
      page %>%
        rvest::html_nodes("div h4 a") %>%
        rvest::html_attr("href")
    },
    error = function(e) {
      print(paste("Error:", conditionMessage(e)))
      return(NULL)
    }
  )
  
  if (!is.null(job_post_links)) {
    print(paste("Scraping URLs from:", url))
    for (link in job_post_links) {
      print(paste("  -", link))
    }
    
    link_texts <- list()
    
    for (link in job_post_links) {
      link_page <- tryCatch(
        {
          read_html(link)
        },
        error = function(e) {
          print(paste("Error accessing URL:", link, "Error:", conditionMessage(e)))
          # 为无法访问的URL添加占位符，保证列表长度匹配
          link_texts[[length(link_texts) + 1]] <- NA
          return(NULL)
        }
      )
      
      if (!is.null(link_page)) {
        link_text <- link_page %>%
          rvest::html_nodes("p") %>%
          rvest::html_text() %>%
          clean_text()
        
        link_texts[[length(link_texts) + 1]] <- link_text
      }
    }
    
    return(link_texts)
  } else {
    return(NULL)
  }
}

内容的提问来源于stack exchange，提问作者Markus Hagedorn