多房产网站数据爬取与CSV合并问题求助（附R代码）

问题诊断 #

1. 函数作用域错误：你将get_reality_idnes嵌套在get_listings内部，导致外部无法调用该函数，运行map_dfr(1:1, get_reality_idnes)会直接报错。
2. HTML选择器不通用：每个房产网站的页面结构完全独立，不能直接复用bezrealitky的选择器，必须针对每个网站单独定位元素。
3. 字段未对齐：现有代码的输出字段和你需求的apartments_pages等目标字段不匹配，需要统一输出结构。

分步实现代码 #

1. 加载依赖包 #

library(tidyverse)
library(rvest)

2. 针对每个网站编写爬取函数 #

每个函数负责爬取单个网站的指定字段，输出统一结构的tibble：

函数1：爬取bezrealitky.cz

get_bezrealitky <- function(page_num) {
  url <- str_c("https://www.bezrealitky.cz/vypis/nabidka-prodej/byt?page=", page_num)
  page <- read_html(url)
  
  listings <- page %>% html_elements(".PropertyCard_propertyCard--disable-link-mask__E6BVo")
  
  tibble(
    apartments_pages = page_num,
    apartments_price = listings %>% 
      html_element(".mb-0.mt-3 span") %>% 
      html_text2(),
    apartments_area = listings %>% 
      html_element(".FeaturesList_featuresListItem__SugGi+ .FeaturesList_featuresListItem__SugGi") %>% 
      html_text2(),
    apartments_floor = listings %>% 
      html_element(".FeaturesList_featuresListItem__SugGi:nth-child(3)") %>% 
      html_text2(),
    apartments_owner = listings %>% 
      html_element(".PropertyCard_supplierInfo__dL8aQ") %>% 
      html_text2(),
    apartments_avaliable = TRUE, # 列表页展示的默认可售
    source = "bezrealitky.cz"
  ) %>% 
    mutate(across(c(apartments_price, apartments_area, apartments_floor), ~str_squish(.x)))
}

函数2：爬取reality.idnes.cz

get_idnes_reality <- function(page_num) {
  url <- str_c("https://reality.idnes.cz/s/prodej/byty/?page=", page_num)
  page <- read_html(url)
  
  listings <- page %>% html_elements(".c-products__item")
  
  tibble(
    apartments_pages = page_num,
    apartments_price = listings %>% 
      html_element(".c-products__price") %>% 
      html_text2(),
    apartments_area = listings %>% 
      html_element(".c-products__info") %>% 
      html_text2() %>% 
      str_extract("\\d+\\s*m²"), # 从info字段提取面积
    apartments_floor = listings %>% 
      html_element(".c-products__info") %>% 
      html_text2() %>% 
      str_extract("\\d+. patro|přízemí"), # 提取楼层信息
    apartments_owner = listings %>% 
      html_element(".c-products__seller") %>% 
      html_text2(),
    apartments_avaliable = TRUE,
    source = "reality.idnes.cz"
  ) %>% 
    mutate(across(c(apartments_price, apartments_area, apartments_floor, apartments_owner), ~str_squish(.x)))
}

函数3：爬取sreality.cz

get_sreality <- function(page_num) {
  url <- str_c("https://www.sreality.cz/hledani/prodej/byty?page=", page_num)
  page <- read_html(url)
  
  listings <- page %>% html_elements(".property")
  
  tibble(
    apartments_pages = page_num,
    apartments_price = listings %>% 
      html_element(".norm-price") %>% 
      html_text2(),
    apartments_area = listings %>% 
      html_element(".property-info .locality") %>% 
      html_text2() %>% 
      str_extract("\\d+\\s*m²"),
    apartments_floor = listings %>% 
      html_element(".property-info .locality") %>% 
      html_text2() %>% 
      str_extract("\\d+. patro|přízemí"),
    apartments_owner = listings %>% 
      html_element(".name") %>% 
      html_text2(),
    apartments_avaliable = TRUE,
    source = "sreality.cz"
  ) %>% 
    mutate(across(c(apartments_price, apartments_area, apartments_floor, apartments_owner), ~str_squish(.x)))
}

3. 批量爬取并合并数据 #

# 定义要爬取的页码范围（示例爬取前2页，可按需调整）
page_range <- 1:2

# 爬取每个网站的数据
data_bezrealitky <- map_dfr(page_range, get_bezrealitky)
data_idnes <- map_dfr(page_range, get_idnes_reality)
data_sreality <- map_dfr(page_range, get_sreality)

# 合并所有数据
combined_data <- bind_rows(data_bezrealitky, data_idnes, data_sreality)

# 保存为CSV文件
write_csv(combined_data, "combined_apartments.csv", na = "")

注意事项 #

• 选择器稳定性：网站的HTML结构可能随时更新，如果后续爬取失败，需要重新用浏览器开发者工具（F12）定位新的元素选择器。
• 反爬机制：部分网站可能有反爬限制，爬取时建议添加延迟（比如用Sys.sleep(2)在每个页面请求后暂停2秒），避免被封禁IP。
• 数据清洗：不同网站的字段格式可能不一致，需要根据实际情况调整str_extract或其他清洗逻辑，确保数据格式统一。

内容的提问来源于stack exchange，提问作者Tomáš Gregorovič