如何用Python+Playwright/Selenium从Onvista提取日内股票数据并修复点击超时

环境与目标 #

• 环境：Python 3.9.6 + Playwright + MacOS 14.7.5
• 目标网页：https://www.onvista.de/aktien/Airbus-Group-EADS-Aktie-NL0000235190
• 问题：点击"Intraday"按钮时触发超时错误Page.click: Timeout 30000ms exceeded，无法加载日内股票数据

原测试脚本 #

import asyncio
from playwright.async_api import async_playwright
from bs4 import BeautifulSoup

async def run():
    url = "https://www.onvista.de/aktien/Airbus-Group-EADS-Aktie-NL0000235190"

    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto(url)

        # Wait for the chart area to appear
        await page.wait_for_selector("svg")

        # Click the 'Intraday' button
        try:
            await page.click("text=Intraday")
            await page.wait_for_timeout(3000)  # wait for chart to update
        except Exception as e:
            print("Couldn't click Intraday:", e)

        # Get the full HTML after the chart is rendered
        content = await page.content()

        # Parse with BeautifulSoup
        soup = BeautifulSoup(content, "html.parser")
        paths = soup.find_all("path")

        print("Extracted SVG path data:")
        for path in paths:
            d_attr = path.get("d")
            if d_attr and len(d_attr) > 100:  # skip small icons etc.
                print("d:", d_attr[:150], "...\n")

        await browser.close()

# Run the async function
asyncio.run(run())

修复Playwright脚本的方法 #

1. 替换文本选择器为更可靠的CSS/XPath选择器 #

文本选择器易受页面语言、元素嵌套影响，改用精准的CSS选择器定位按钮：

# 替换原点击Intraday的代码块
try:
    # 等待按钮可见并可点击
    intraday_btn = await page.wait_for_selector('button:has-text("Intraday")', state="enabled", timeout=15000)
    await intraday_btn.click()
    # 等待图表数据加载
    await page.wait_for_selector('svg path[d]', state="attached", timeout=10000)
except Exception as e:
    print("Couldn't click Intraday:", e)

也可使用XPath定位：

intraday_btn = await page.wait_for_selector('//button[contains(text(), "Intraday")]', state="visible")

2. 处理Cookie弹窗（页面加载优先级问题） #

页面加载时可能弹出Cookie授权窗口，遮挡或阻塞后续元素交互，需先处理：

# 在page.goto之后添加Cookie弹窗处理
await page.goto(url)
# 尝试关闭Cookie同意弹窗
try:
    await page.click('button:has-text("Akzeptieren")', timeout=5000)
except:
    pass  # 无弹窗则忽略

3. 临时关闭无头模式调试 #

关闭无头模式直观查看页面状态，确认按钮是否存在、是否被遮挡：

browser = await p.chromium.launch(headless=False)

4. 优化等待逻辑，替换固定超时 #

避免使用固定wait_for_timeout，改为等待图表元素更新：

await intraday_btn.click()
# 等待日内图表的SVG路径加载完成
await page.wait_for_function('document.querySelectorAll("svg path[d]").length > 10')

Selenium是否是更好的选择？ #

Playwright在自动等待、API简洁性上更有优势，当前问题仅需调整选择器和等待逻辑即可修复，无需切换到Selenium。除非后续遇到旧浏览器兼容等特殊场景，否则优先优化Playwright脚本更高效。

内容的提问来源于stack exchange，提问作者Alex