使用Python BeautifulSoup爬取Zillow租房数据遇空结果求助

问题根源 #

你之前的代码拿不到数据，核心原因是Zillow的房源内容是JavaScript动态渲染的，BeautifulSoup直接解析静态HTML只能拿到空的容器，无法获取实际加载的房源数据。

正确实现方法（Selenium + BeautifulSoup） #

1. 依赖安装 #

先安装所需工具库：

pip install selenium beautifulsoup4 webdriver-manager

2. 完整代码实现 #

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from bs4 import BeautifulSoup
import time

# 初始化Chrome驱动
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
target_url = "https://www.zillow.com/homes/for_rent/?searchQueryState=%7B%22pagination%22%3A%7B%7D%2C%22mapBounds%22%3A%7B%22west%22%3A-122.48906135559082%2C%22east%22%3A-122.35413551330566%2C%22south%22%3A37.738955793388136%2C%22north%22%3A37.82727700595516%7D%2C%22mapZoom%22%3A13%2C%22isMapVisible%22%3Afalse%2C%22filterState%22%3A%7B%22price%22%3A%7B%22min%22%3A0%2C%22max%22%3A872627%7D%2C%22beds%22%3A%7B%22min%22%3A1%7D%2C%22fore%22%3A%7B%22value%22%3Afalse%7D%2C%22mp%22%3A%7B%22min%22%3A0%2C%22max%22%3A3000%7D%2C%22ah%22%3A%7B%22value%22%3Atrue%7D%2C%22auc%22%3A%7B%22value%22%3Afalse%7D%2C%22nc%22%3A%7B%22value%22%3Afalse%7D%2C%22fr%22%3A%7B%22value%22%3Atrue%7D%2C%22fsbo%22%3A%7B%22value%22%3Afalse%7D%2C%22cmsn%22%3A%7B%22value%22%3Afalse%7D%2C%22fsba%22%3A%7B%22value%22%3Afalse%7D%7D%2C%22isListVisible%22%3Atrue%7D"

# 打开页面并等待渲染完成
driver.get(target_url)
time.sleep(3)  # 可根据网络速度调整等待时长

# 获取渲染后的完整HTML并解析
soup = BeautifulSoup(driver.page_source, 'html.parser')
driver.quit()  # 关闭浏览器

# 定位目标房源卡片元素
property_cards = soup.find_all('div', class_='StyledPropertyCardDataWrapper-c11n-8-84-3__sc-1omp4c3-0 bKpguY property-card-data')

# 初始化三个存储数据的列表
prices = []
addresses = []
links = []

# 遍历卡片提取数据
for card in property_cards:
    # 提取价格
    price_elem = card.find('span', {'data-test': 'property-card-price'})
    prices.append(price_elem.get_text(strip=True) if price_elem else None)
    
    # 提取地址
    addr_elem = card.find('address', {'data-test': 'property-card-addr'})
    addresses.append(addr_elem.get_text(strip=True) if addr_elem else None)
    
    # 提取链接并补全域名
    link_elem = card.find_parent('a', {'data-test': 'property-card-link'})
    if link_elem and 'href' in link_elem.attrs:
        full_link = link_elem['href'] if link_elem['href'].startswith('http') else f"https://www.zillow.com{link_elem['href']}"
        links.append(full_link)
    else:
        links.append(None)

# 输出示例结果
print("价格列表（前3条）:", prices[:3])
print("地址列表（前3条）:", addresses[:3])
print("链接列表（前3条）:", links[:3])

3. 关键说明 #

• 动态渲染处理：用Selenium模拟真实浏览器加载页面，确保获取到JS渲染后的完整房源数据。
• 稳定定位：优先使用data-test属性提取价格、地址，避免网站更新类名导致定位失效。
• 链接补全：Zillow房源链接多为相对路径，需补全域名得到完整可访问的URL。

注意事项 #

• Zillow有反爬机制，频繁请求可能触发IP封禁，建议添加随机延迟、使用代理IP。
• 若后续页面结构更新，需重新检查元素定位规则。

内容的提问来源于stack exchange，提问作者LiorMorali