使用rvest爬取VISA西班牙ATM定位器数据失败的技术求助

Hey Diana, let's break down why your current code isn't pulling the ATM data you want, and walk through a working solution to get that Spanish ATM list in a table!

The VISA ATM Locator page is a JavaScript-rendered single-page application. When you use read_html() directly, you're only fetching the static base HTML of the page—none of the ATM list content has loaded yet, because that data is pulled dynamically by JavaScript after the initial page loads. That's why html_nodes("visaATMResultListItem") returns nothing.

To get the dynamically loaded content, we need to mimic a real user browsing the page. RSelenium lets us launch a browser, load the page (including executing all JavaScript), and then grab the fully rendered HTML.

Step 1: Install Required Packages #

First, install the tools we'll need:

install.packages(c("RSelenium", "rvest", "dplyr", "stringr"))

Step 2: Full Working Code #

This code will launch Chrome, load the page, extract the ATM data, and format it into a table:

library(RSelenium)
library(rvest)
library(dplyr)
library(stringr)

# Launch Selenium and Chrome (make sure ChromeDriver matches your Chrome version)
# Replace the chromever value with your exact Chrome browser version
driver <- rsDriver(browser = "chrome", chromever = "118.0.5993.70")
remDr <- driver$client

# Navigate to the page and wait for dynamic content to load
remDr$navigate("https://www.visa.com/atmlocator/mobile/index.jsp#(page:results,params:(query:Spain))")
Sys.sleep(10) # Adjust this based on your internet speed (longer if slow)

# Grab the fully rendered page source
page_source <- remDr$getPageSource()[[1]]
visa_webpage <- read_html(page_source)

# Extract ATM elements and parse their details
atm_items <- visa_webpage %>% html_nodes(".visaATMResultListItem")

# Build a data frame/table with the ATM info
atm_table <- tibble(
  Location_Name = atm_items %>% html_node(".locationName") %>% html_text(trim = TRUE),
  Full_Address = atm_items %>% html_node(".locationAddress") %>% html_text(trim = TRUE) %>% str_replace_all("\\n", ", "),
  City = atm_items %>% html_node(".locationCity") %>% html_text(trim = TRUE),
  Distance = atm_items %>% html_node(".locationDistance") %>% html_text(trim = TRUE)
)

# View the final table
print(atm_table)

# Clean up: Close the browser and stop Selenium
remDr$close()
driver$server$stop()

Important Notes #

• ChromeDriver Version Match: This is critical! Check your Chrome version (Settings > About Chrome) and use the matching ChromeDriver version in the chromever parameter. If they don't match, the browser won't launch.
• Wait Time: Sys.sleep(10) gives the page time to load all ATM data. If you're on a slow connection, increase this number. For a more robust approach, use remDr$waitForElement(using = "css", value = ".visaATMResultListItem", timeout = 30000) to wait until the first ATM item appears (timeout is in milliseconds).
• HTML Structure Changes: If VISA updates their page layout, you'll need to adjust the CSS selectors (like .locationName). Use your browser's DevTools (F12) to inspect the elements and find the correct selectors.

内容的提问来源于stack exchange，提问作者Diana