如何用XPath抓取JS渲染网站数据？以美国运通储蓄利率为例

一、先修复你当前代码的核心问题 #

你当前的代码存在硬编码元素ID、未等待页面加载、反爬检测触发三个主要问题，先调整为可运行的版本：

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

service = Service(executable_path="/chromedriver")
options = Options()
# 修复参数格式+添加反爬规避配置
options.add_argument('--incognito')
options.add_argument('--disable-blink-features=AutomationControlled')
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
# 优先用非无头模式测试，必要时再开启并补充UA等参数
# options.headless = True

driver = webdriver.Chrome(service=service, options=options)

url = 'https://www.americanexpress.com/en-us/banking/online-savings/account/'
driver.get(url)

try:
    # 等待元素加载（最多10秒），用特征匹配替代硬编码ID
    element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.XPATH, '//*[contains(text(), "%") and contains(text(), "APY")]'))
    )
    print("抓取到的利率：", element.text)
except Exception as e:
    print("抓取失败：", str(e))
    # 排查时可打印页面源码：print(driver.page_source)

driver.quit()

二、通用XPath变量设计思路 #

储蓄利率有明确的共性特征，基于这些特征设计可复用的XPath模板，就能适配绝大多数同类站点：

• 核心特征：包含百分比符号(%)、APY/Yield等关键词，常出现在强调文本、标题或带特定标识的容器中
• 通用XPath模板示例：
  • 忽略大小写匹配APY+百分比：//*[contains(text(), "%") and contains(translate(text(), "APY", "apy"), "apy")]
  • 匹配带rate/apy类名的元素：//*[contains(@class, "rate") or contains(@class, "apy")]
  • 匹配标题标签中的利率：//h1|h2|h3|h4[contains(text(), "%")]
  • 匹配高亮样式的利率：//span[contains(@class, "highlight") and contains(text(), "%")]

三、多站点通用抓取框架 #

把XPath模板做成可配置列表，遍历尝试直到抓取到有效数据，实现一站适配多站点：

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

def scrape_savings_rate(url, xpath_templates):
    service = Service(executable_path="/chromedriver")
    options = Options()
    options.add_argument('--incognito')
    options.add_argument('--disable-blink-features=AutomationControlled')
    options.add_experimental_option("excludeSwitches", ["enable-automation"])
    options.add_experimental_option('useAutomationExtension', False)
    
    driver = webdriver.Chrome(service=service, options=options)
    driver.get(url)
    rate = None
    
    try:
        for xpath in xpath_templates:
            try:
                element = WebDriverWait(driver, 8).until(
                    EC.presence_of_element_located((By.XPATH, xpath))
                )
                text = element.text.strip()
                # 验证是否为有效利率（必须包含%）
                if "%" in text:
                    rate = text
                    break
            except:
                continue
    except Exception as e:
        print(f"页面加载失败：{str(e)}")
    finally:
        driver.quit()
    
    return rate if rate else "未抓取到有效利率"

# 通用XPath模板集合，可根据新站点特征补充
COMMON_XPATH_TEMPLATES = [
    '//*[contains(text(), "%") and contains(translate(text(), "APY", "apy"), "apy")]',
    '//*[contains(@class, "rate") or contains(@class, "apy")]',
    '//*[contains(text(), "Yield") and contains(text(), "%")]',
    '//h1|h2|h3|h4[contains(text(), "%")]',
    '//span[contains(@class, "highlight") and contains(text(), "%")]'
]

# 测试美国运通站点
url = 'https://www.americanexpress.com/en-us/banking/online-savings/account/'
result = scrape_savings_rate(url, COMMON_XPATH_TEMPLATES)
print(f"储蓄利率：{result}")

四、关键注意事项 #

• 反爬规避：添加随机User-Agent、请求间隔，避免直接使用无头模式；必要时搭配代理IP
• 高效替代方案：部分站点的利率通过AJAX接口返回，可通过浏览器开发者工具抓包获取接口，直接用requests请求比Selenium更高效
• 模板扩展：遇到特殊站点时，补充该站点专属的XPath到模板列表，进一步提升抓取命中率

内容的提问来源于stack exchange，提问作者jasonlovesplanes