使用Playwright Python爬取弹窗建筑年份并存储到Pandas DataFrame

问题背景 #

我在Jupyter Notebook中使用Playwright爬取房产评估网站数据，部分页面需点击View Building Details按钮才能查看多建筑信息。目前已实现点击按钮并切换至Building 2、3、4，但始终无法提取Structural Element for Building ###表格下的Year Built数据。尝试过Pandas的read_html、Playwright的wait_for_selector、wait_for_load_state("networkidle")等方法，仍存在加载相关问题。

解决方案 #

1. 精准等待目标元素，而非仅依赖网络状态 #

networkidle仅代表网络请求暂停，但动态渲染的表格可能仍未完成DOM插入。需直接等待Year Built所在的表格行出现，确保元素可被定位。

2. 使用更精准的元素选择器 #

避免用宽泛的表格选择器，结合表格标题和行文本定位目标单元格，比如用XPath关联表格标题与目标行，避免误定位其他建筑的表格。

3. 切换建筑后等待元素更新 #

切换建筑标签后页面通常是局部刷新，需等待旧表格元素消失、新表格元素加载完成，防止读取缓存的旧数据。

代码示例 #

from playwright.sync_api import sync_playwright
import pandas as pd

def extract_building_year(page, building_num):
    # 切换到目标建筑标签
    page.click(f"text=Building {building_num}")
    
    # 等待目标表格标题加载
    table_title_selector = f"h3:text('Structural Element for Building {building_num}')"
    page.wait_for_selector(table_title_selector, timeout=10000)
    
    # 用XPath定位Year Built对应的数值单元格
    year_xpath = (
        f"//h3[contains(text(), 'Structural Element for Building {building_num}')]"
        "/following-sibling::table//tr[td[normalize-space(text())='Year Built']]/td[2]"
    )
    year_element = page.wait_for_selector(year_xpath, timeout=10000)
    
    return year_element.inner_text().strip()

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)
    page = browser.new_page()
    target_url = "你的目标房产页面URL"
    page.goto(target_url)
    
    # 展开建筑详情
    page.click("text=View Building Details")
    page.wait_for_load_state("domcontentloaded")
    
    # 获取总建筑数量（通过标签数量判断）
    building_tab_selector = "button:text-matches('Building \\d+')"
    page.wait_for_selector(building_tab_selector)
    total_buildings = len(page.query_selector_all(building_tab_selector))
    
    # 循环提取Building 2到n的年份
    for num in range(2, total_buildings + 1):
        try:
            built_year = extract_building_year(page, num)
            print(f"Building {num} 建造年份: {built_year}")
        except Exception as e:
            print(f"提取Building {num} 年份失败: {str(e)}")
    
    browser.close()

补充技巧 #

• 监听AJAX请求：如果表格数据通过接口加载，可通过page.wait_for_response等待目标接口返回：

  response = page.wait_for_response(lambda r: "building-details" in r.url and r.status == 200)
  data = response.json()
  # 从返回的JSON中直接提取年份
  

• 结合Pandas处理HTML：若仍想用read_html，需先获取当前页面的完整DOM：

  page.wait_for_selector("text=Year Built")
  html_content = page.content()
  dfs = pd.read_html(html_content)
  for df in dfs:
      if "Year Built" in df.iloc[:,0].values:
          year = df[df.iloc[:,0] == "Year Built"].iloc[0,1]
          print(year)
          break
  

• Jupyter环境注意事项：优先使用Playwright同步模式，避免异步模式与Jupyter事件循环冲突；若用异步，需在单元格开头加%asyncio_mode=strict。

内容的提问来源于stack exchange，提问作者Adam