使用R与RSelenium批量爬取谷歌评论时重复数据问题排查

我正在开发一个循环脚本，批量在谷歌搜索多家公司，点击「More Reviews」链接加载全部评论。目前已实现滚动到评论底部并开始抓取信息，但评论内容（rev_text）和评分（rev_score）总是出现重复数据，试了多种XPath组合都没解决。当前用单公司测试，后续会改成遍历公司列表的循环。

示例测试公司：

company <- "5280 Trees"

当前执行流程代码 #

# Navigate to Google
remDr$navigate("https://www.google.com")

# Find the search box and enter the company name
search_box <- remDr$findElement(using = "name", value = "q") 
search_box$sendKeysToElement(list(company, key = "enter"))
# Select the review dialog for scrolling
select_review_dialog <- remDr$findElement(using = "class", value = "review-dialog-list")

# Scroll within the review dialog
for (i in 1:10) {  # Adjust the loop count as needed
  remDr$executeScript("arguments[0].scrollTop = arguments[0].scrollHeight", list(select_review_dialog))
  Sys.sleep(1)
}

Sys.sleep(2)  # Pause to let the page load

# Identify and click on the more reviews button  
more_reviews_button <- remDr$findElement(using = "xpath", value = "//span[contains(text(),'Google reviews')]")
more_reviews_button$clickElement()
Sys.sleep(2)  # Pause to let the reviews load

# Select the review dialog for scrolling
select_review_dialog <- remDr$findElement(using = "class", value = "review-dialog-list")

# Scroll within the review dialog
for (i in 1:10) {  # Adjust the loop count as needed
  remDr$executeScript("arguments[0].scrollTop = arguments[0].scrollHeight", list(select_review_dialog))
  Sys.sleep(1)
}

当前抓取情况 #

评论者姓名（无重复，正常获取） #

抓取代码：

# Scrape for reviewer name 
rev_name <- read_html(remDr$getPageSource()[[1]]) %>% 
  html_elements(".TSUbDb a") %>% 
  html_text() %>% 
  tibble() %>% 
  rename(reviewer_name = ".")

结果示例：

A tibble:77 × 1
reviewer_name
<chr>
Joshua Riehl                
Sean O'Connor               
Lynn Dexter              
Willy               
James Paris              
Eric Aitken              
Alan Kamp               
Suzanne Cook                
D RDH               
Kirsten Smith

评论内容（大量重复） #

抓取代码：

# Scrape for review text
rev_text <- read_html(remDr$getPageSource()[[1]]) %>% 
  html_elements(".Jtu6Td span") %>% 
  html_text() %>% 
  tibble() %>% 
  rename(text = ".")

重复结果示例：

A tibble:420 × 1
text
<chr>
I recently had the pleasure of working with 5280 Trees after contacting multiple tree service companies, and I can confidently say they exceeded my expectations in every way. Out of the 7 companies I reached out to, only three responded, and 5280 Trees stood out as the most professional and fairly priced.They removed two large trees from a couple of my rental properties, and even my next-door neighbor commented on how professional the crew was. I initially expected the job to take half a day due to the size of the trees, but the four-member team had everything completed and cleaned up in just two hours.It’s rare these days to find companies that truly deliver value, where you feel like you’re getting more for what you pay. With 5280 Trees, I felt like I paid less and received more. Their equipment was top-notch, and the crew was courteous, efficient, and left the property spotless.I wouldn’t hesitate to use 5280 Trees again, and I highly recommend them to anyone looking for professional, reliable tree services.I recently had the pleasure of working with 5280 Trees after contacting multiple tree service companies, and I can confidently say they exceeded my expectations in every way. Out of the 7 companies I reached out to, only three responded, …More
I recently had the pleasure of working with 5280 Trees after contacting multiple tree service companies, and I can confidently say they exceeded my expectations in every way. Out of the 7 companies I reached out to, only three responded, and 5280 Trees stood out as the most professional and fairly priced.They removed two large trees from a couple of my rental properties, and even my next-door neighbor commented on how professional the crew was. I initially expected the job to take half a day due to the size of the trees, but the four-member team had everything completed and cleaned up in just two hours.It’s rare these days to find companies that truly deliver value, where you feel like you’re getting more for what you pay. With 5280 Trees, I felt like I paid less and received more. Their equipment was top-notch, and the crew was courteous, efficient, and left the property spotless.I wouldn’t hesitate to use 5280 Trees again, and I highly recommend them to anyone looking for professional, reliable tree services.I recently had the pleasure of working with 5280 Trees after contacting multiple tree service companies, and I can confidently say they exceeded my expectations in every way. Out of the 7 companies I reached out to, only three responded, …More
I recently had the pleasure of working with 5280 Trees after contacting multiple tree service companies, and I can confidently say they exceeded my expectations in every way. Out of the 7 companies I reached out to, only three responded, and 5280 Trees stood out as the most professional and fairly priced.They removed two large trees from a couple of my rental properties, and even my next-door neighbor commented on how professional the crew was. I initially expected the job to take half a day due to the size of the trees, but the four-member team had everything completed and cleaned up in just two hours.It’s rare these days to find companies that truly deliver value, where you feel like you’re getting more for what you pay. With 5280 Trees, I felt like I paid less and received more. Their equipment was top-notch, and the crew was courteous, efficient, and left the property spotless.I wouldn’t hesitate to use 5280 Trees again, and I highly recommend them to anyone looking for professional, reliable tree services.

评分（大量重复） #

抓取代码：

#scrape for review score
rev_score <- read_html(remDr$getPageSource()[[1]]) %>% 
  html_elements(".z3HNkc") %>% 
  html_attr("aria-label") %>% 
  tibble() %>% 
  rename(score = ".")

重复结果示例：

A tibble:157 × 1
score
<chr>
Rated 4.6 out of 5,             
Rated 4.6 out of 5,             
Rated 4.7 out of 5,             
Rated 5.0 out of 5,             
Rated 5.0 out of 5,             
Rated 5.0 out of 5,             

需要解决的问题 #

• 修复评论内容和评分的重复数据问题，或提供一次性关联抓取评论者姓名、评论内容、评分的方法
• 实现抓取评论发布日期
• 在循环中自动跳过没有「更多评论」选项的公司

内容的提问来源于stack exchange，提问作者Ben