在R语言中遍历XPath节点时出现NA值问题求助

Hey there! Let's tackle that NA issue you're hitting when scraping Zap Imoveis in R with XPath. I've dealt with similar headaches on real estate sites before—they often have inconsistent node structures (some listings skip fields like parking spots or balcony details), which is why those NAs are popping up. Here's how to fix it step by step:

The biggest mistake when scraping these sites is trying to extract all fields at once with html_nodes(). If even one listing is missing a field, you'll get mismatched lengths and random NAs. Instead, target each listing container first, then extract fields from each container individually.

Example code:

library(rvest)
library(purrr)
library(dplyr)

# First, grab the container for every listing (update XPath to match Zap's current structure)
listing_containers <- read_html("your-zap-url") %>% 
  html_nodes(xpath = "//div[contains(@class, 'card__content')]")

# Define a function to extract data from a single listing container
extract_listing <- function(container) {
  # Extract price (use html_node() instead of html_nodes() to get 1 result per listing)
  price <- container %>% 
    html_node(xpath = ".//span[contains(@class, 'price__main')]") %>% 
    html_text(trim = TRUE) %>% 
    ifelse(is.na(.), NA, .)
  
  # Extract area
  area <- container %>% 
    html_node(xpath = ".//li[contains(@class, 'feature__item--area')]") %>% 
    html_text(trim = TRUE) %>% 
    ifelse(is.na(.), NA, .)
  
  # Extract bedroom count
  bedrooms <- container %>% 
    html_node(xpath = ".//li[contains(@class, 'feature__item--bedrooms')]") %>% 
    html_text(trim = TRUE) %>% 
    ifelse(is.na(.), NA, .)
  
  # Return a tibble row for the listing
  tibble(price = price, area = area, bedrooms = bedrooms)
}

# Apply the function to all containers and combine into a data frame
listing_data <- map_dfr(listing_containers, extract_listing)

Zap Imoveis loads most listings via JavaScript—if you're using read_html() directly, you're only grabbing the initial static HTML, not the fully loaded listings. That's why you might see way more NAs than expected.

Fix this with one of two approaches:

• Use RSelenium to simulate a browser:

  library(RSelenium)
  
  # Start a Chrome session (make sure ChromeDriver is installed)
  driver <- rsDriver(browser = "chrome", port = 4567L)
  remDr <- driver[["client"]]
  
  # Navigate to the page and wait for JS to load
  remDr$navigate("https://www.zapimoveis.com.br/aluguel/imoveis/rj+rio-de-janeiro/")
  Sys.sleep(5) # Adjust delay based on your internet speed
  
  # Grab the fully loaded page source
  page_source <- remDr$getPageSource()[[1]]
  page <- read_html(page_source)
  
  # Now extract listings using the method from step 1...
  
  # Clean up
  remDr$close()
  driver$server$stop()
  

• Scrape the underlying API: Most real estate sites pull data from a JSON API. Use browser dev tools (Network tab) to find the API endpoint, then use httr to fetch the data directly—this avoids HTML parsing issues entirely.

Once you have your raw data, you can handle remaining NAs based on your needs:

# Remove rows where all fields are NA (likely failed extractions)
clean_data <- listing_data %>% 
  filter(!if_all(everything(), is.na()))

# Replace NAs with a human-readable placeholder
filled_data <- listing_data %>% 
  replace_na(list(price = "Not listed", area = "Not specified", bedrooms = "Not specified"))

内容的提问来源于stack exchange，提问作者lucasm