Python Playwright爬取律师页下拉选项选择操作问题求解

现有代码无法选中每页展示条数下拉选项，由4个问题导致：

• 选项匹配方式错误：使用{"label": "30"}匹配下拉选项，实际ASP.NET渲染的option标签文本可能带不可见空白字符，匹配极易失败，直接匹配选项的value属性稳定性更高
• 操作时机错误：在每一页的循环中重复执行下拉选择操作，该下拉框是服务器端控件，选中后会触发全页postback刷新，重复触发会打乱页面状态，导致翻页、选值逻辑异常，该操作仅需在首次进入页面时执行1次
• 等待逻辑不可靠：全部使用time.sleep()硬等待，既无法保证postback完成后再执行后续操作，也会拖慢爬取效率，应该使用Playwright内置的加载状态判断
• 浏览器内核兼容问题：使用webkit内核启动，该网站对webkit的表单交互适配存在bug，切换为chromium内核可大幅提升稳定性

1. 页面首次加载完成后，仅执行1次下拉选择操作，通过value属性选中30条/页的选项
2. 下拉选值、点击翻页后，用wait_for_load_state("networkidle")等待页面请求完成、内容渲染完毕，替代硬等待
3. 替换webkit为chromium内核启动浏览器
4. 去掉硬编码的2450次循环，改为判断下一页按钮是否可用来终止循环，避免最后一页触发无效点击报错
5. 导出csv时增加index=False参数，去除pandas自动生成的无用索引列

from playwright.sync_api import sync_playwright
import pandas as pd

with sync_playwright() as p:
    browser = p.chromium.launch(headless=False)
    baseurl = "https://www.ifep.ro/justice/lawyers/lawyerspanel.aspx"
    page = browser.new_page()
    page.goto(baseurl, wait_until="networkidle")
    
    # 首次进入页面设置每页展示30条，通过value匹配选项
    page.wait_for_selector("#MainContent_ddlRecords")
    page.select_option("#MainContent_ddlRecords", value="30")
    # 等待选值触发的页面刷新完成
    page.wait_for_load_state("networkidle")
    
    productlinks = []
    # 循环翻页直到最后一页
    while True:
        page.wait_for_selector("div.list-group a")
        links = page.query_selector_all("//div[@class='list-group']//a")
        for link in links:
            link_href = link.get_attribute("href")
            if link_href.startswith("LawyerFile.aspx"):
                productlinks.append("https://www.ifep.ro/justice/lawyers/" + link_href)
        
        next_btn = page.query_selector("#MainContent_PagerTop_NavNext")
        # 下一页按钮禁用说明到达最后一页，退出循环
        if not next_btn or next_btn.is_disabled():
            break
        
        next_btn.click()
        page.wait_for_load_state("networkidle")
    
    data = []    
    for product in productlinks:
        wev = {}
        page.goto(product, wait_until="networkidle")
        wev['title'] = page.wait_for_selector('#HeadingContent_lblTitle').text_content()
        d1 = page.wait_for_selector("//div[@class='col-md-10']//p[1]").text_content().strip().split()[-1]
        wev['Avocat Definitiv'] = d1
        wev['Email'] = page.wait_for_selector("//span[@class='text-nowrap']//a").text_content()
        d5 = page.wait_for_selector("//span[@class='padding-right-md text-primary']").text_content().replace(".", "")
        wev['phone'] = d5
        d2 = page.wait_for_selector("//div[@class='col-md-10']//p[2]").text_content().strip().split()[-1]
        wev['Dată înscriere'] = d2
        d3 = page.wait_for_selector("//div[@class='col-md-10']//p[3]//span").text_content().strip().split()[-1]
        wev['Situaţie curentă în tablou'] = d3
        d4 = page.wait_for_selector("//div[@class='col-md-10']//p[4]").text_content().strip().split()[-1]
        wev['Instanţe cu drept de concluzii'] = d4
        data.append(wev) 
        
    df = pd.DataFrame(data)
    df.to_csv("test.csv", index=False)
    browser.close()

• 如果需要切换其他每页展示条数，直接修改select_option的value参数即可，可选值为"10"、"30"、"50"、"100"
• 爬取详情页如果出现超时，可给goto方法增加timeout=60000参数，将超时时间设置为60秒
• 如果需要无头运行，把headless=False改成headless=True即可

内容的提问来源于stack exchange，提问作者Amen Aziz