Zillow爬虫问题：使用Python requests无法爬取全部房源列表如何解决？

问题核心原因 #

你当前的代码存在4个核心问题导致爬取数据量少：

• 你自定义的searchQueryState中mapBounds经纬度范围远小于实际搜索页面的范围，仅覆盖了目标区域的一小部分，自然只能拿到部分房源
• Zillow当前版本的页面仅会在静态HTML中渲染前9-11条房源，剩余房源数据存储在页面内置的__NEXT_DATA__脚本节点中，直接通过DOM节点抓取只能拿到首屏数据
• 分页逻辑错误，你当前循环仅执行了1次（range(1,2)仅取到1），只爬取了第一页数据
• 请求头配置不全，缺少必要的referer等字段，容易被Zillow的反爬策略返回阉割版页面

具体修改方案 #

1. 用字典构造searchQueryState参数再转成JSON字符串，避免手动拼接字符串出现语法错误，同时还原和原页面一致的mapBounds范围
2. 直接提取页面__NEXT_DATA__节点的JSON数据，而非抓取DOM元素，可拿到当前页全部房源信息
3. 补全请求头，使用requests.Session()维持会话，降低被反爬识别的概率
4. 修正分页逻辑，根据返回的房源数量判断是否停止翻页，同时添加请求间隔避免触发频率限制

修复后的代码 #

import requests
import json
import time
from bs4 import BeautifulSoup as soup
import pandas as pd

# 初始化会话
session = requests.Session()
all_houses = []
base_url = 'https://www.zillow.com/birmingham-al-35216/'
headers = {
    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
    'accept-encoding': 'gzip, deflate, br',
    'accept-language': 'en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7,zh-TW;q=0.6',
    'referer': 'https://www.zillow.com/',
    'upgrade-insecure-requests': '1',
    'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
}
session.headers.update(headers)

page_num = 1
empty_count = 0
# 最大分页防止死循环，76条最多翻8页
max_page = 10

while page_num <= max_page and empty_count < 3:
    # 用字典构造参数，避免拼接错误
    query_state = {
        "pagination": {"currentPage": page_num},
        "usersSearchTerm": "35216",
        "mapBounds": {
            "west": -86.93997505787829,
            "east": -86.62926796559313,
            "south": 33.33562772711966,
            "north": 33.51819716059094
        },
        "regionSelection": [{"regionId": 73386, "regionType": 7}],
        "isMapVisible": True,
        "filterState": {
            "sort": {"value": "globalrelevanceex"},
            "ah": {"value": True}
        },
        "isListVisible": True,
        "mapZoom": 13
    }
    params = {'searchQueryState': json.dumps(query_state)}
    
    try:
        resp = session.get(base_url, params=params, timeout=10)
        resp.raise_for_status()
        sp = soup(resp.text, 'lxml')
        
        # 提取__NEXT_DATA__里的完整房源数据
        next_data_script = sp.find('script', id='__NEXT_DATA__')
        if not next_data_script:
            empty_count +=1
            page_num +=1
            time.sleep(3)
            continue
        
        next_data = json.loads(next_data_script.contents[0])
        house_list = next_data['props']['pageProps']['searchPageState']['cat1']['searchResults']['listResults']
        
        if not house_list:
            empty_count +=1
        else:
            empty_count = 0
            all_houses.extend(house_list)
            print(f"第{page_num}页，获取到{len(house_list)}条房源，累计{len(all_houses)}条")
        
        page_num +=1
        # 加请求间隔避免被封
        time.sleep(3)
    
    except Exception as e:
        print(f"第{page_num}页请求出错: {str(e)}")
        time.sleep(5)
        continue

# 输出结果到CSV
df = pd.DataFrame(all_houses)
df.to_csv('zillow_35216_houses.csv', index=False, encoding='utf-8-sig')
print(f"总共获取到{len(all_houses)}条房源数据")

补充说明 #

如果修改后仍存在数据不全的情况，可改用无头浏览器（如Playwright、Selenium）模拟真实用户浏览行为，和你使用的Chrome插件原理一致，可完全规避静态页面抓取的限制。

内容的提问来源于stack exchange，提问作者Yuhang Wu