如何在Python版Selenium 4中从JavaScript提取HTML？

问题背景 #

在Python版Selenium 4中尝试提取JavaScript渲染生成的HTML时，发现Selenium 3.8里可用的PhantomJS方法已被废弃，执行代码时反复报错：

AttributeError: module 'selenium.webdriver' has no attribute 'PhantomJS'

现有大量Selenium资料仍在使用该已不支持的功能，需确认是否存在可行的替代方案实现目标。

验证页面需JavaScript渲染的代码 #

以下代码用于验证目标页面内容由JavaScript动态生成（直接请求仅能获取空模板HTML）：

# Libraries
import urllib3
from bs4 import BeautifulSoup
from time import sleep

# Wait 3 Seconds
sleep(3)

# Store URL
url = "https://beta.goal3.xyz/sports/soccer"

# Define the headers
headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/113.0.5672.127 Safari/537.11',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
    'Accept-Encoding': 'none',
    'Accept-Language': 'en-US,en;q=0.8',
    'Connection': 'keep-alive'
}

# Create an instance of the PoolManager with the specified headers
http = urllib3.PoolManager(headers=headers)

# Send a GET request to the website
response = http.request('GET', url)

# Check if the request was successful
if response.status == 200:
    # Parse the HTML content using BeautifulSoup
    soup = BeautifulSoup(response.data, 'html.parser')
    print(soup)

目标页面原始HTML #

直接请求获取的页面仅包含模板结构，实际内容由JavaScript渲染：

<!DOCTYPE html>
<html class="dark" data-theme="dark" lang="en"><title>Goal3.xyz - The first decentralized Sportsbook on zkSync Era</title><head><meta charset="utf-8"/><meta content="viewport-fit=cover width=device-width, initial-scale=1, maximum-scale=1" name="viewport"/><meta content="2" name="next-head-count"/><script async="" data-id="101408729" src="//static.getclicky.com/js"></script><noscript><p><img alt="Clicky" height="1" src="//in.getclicky.com/101408729ns.gif" width="1"/></p></noscript><meta content="yes" name="apple-mobile-web-app-capable"/><meta content="yes" name="mobile-web-app-capable"/><meta content="Experience the Future of Sport Betting with Goal3 - Powered by zkEVM. Join the GambleFi movement and unleash the power of decentralized betting!" name="description"/><meta content="sportsbook, zkSync, betting, web3, blockchain." itemprop="keywords" name="keywords"/><meta name="news_keywords"/><meta content="Goal3" property="og:site_name"/><meta content="website" property="og:type"/><meta content="https://beta.goal3.xyz" property="og:url"/><meta content="https://beta.goal3.xyz/meta/default.jpg" property="og:image"/><meta content="image/png" property="og:image:type"/><link as="style" href="/_next/static/css/f727b947152586c6.css" rel="preload"/><link data-n-g="" href="/_next/static/css/f727b947152586c6.css" rel="stylesheet"/><link as="style" href="/_next/static/css/361e7b7ce229ef67.css" rel="preload"/><link data-n-p="" href="/_next/static/css/361e7b7ce229ef67.css" rel="stylesheet"/><link as="style" href="/_next/static/css/eb36c408c8c0fcf5.css" rel="preload"/><link data-n-p="" href="/_next/static/css/eb36c408c8c0fcf5.css" rel="stylesheet"/><noscript data-n-css=""></noscript><script defer="" nomodule="" src="/_next/static/chunks/polyfills-c67a75d1b6f99dc8.js"></script><script defer="" src="/_next/static/chunks/webpack-aa0d6a8884a8520d.js"></script><script defer="" src="/_next/static/chunks/framework-ce84985cd166733a.js"></script><script defer="" src="/_next/static/chunks/main-354042be1614f0c6.js"></script><script defer="" src="/_next/static/chunks/pages/_app-3135abef21e8aa92.js"></script><script defer="" src="/_next/static/chunks/75fc9c18-0061c64dcc709f97.js"></script><script defer="" src="/_next/static/chunks/24-67cd99fa99f141f2.js"></script><script defer="" src="/_next/static/chunks/1664-6077f61cd231cbb6.js"></script><script defer="" src="/_next/static/chunks/719-cd8014cbd98dd40e.js"></script><script defer="" src="/_next/static/chunks/6898-545aecb4c6bb194d.js"></script><script defer="" src="/_next/static/chunks/6554-1aa0abaadf7beac8.js"></script><script defer="" src="/_next/static/chunks/8208-8e5070115ebb7620.js"></script><script defer="" src="/_next/static/chunks/683-c2d3bba216c08d07.js"></script><script defer="" src="/_next/static/chunks/4159-9547181b9f859321.js"></script><script defer="" src="/_next/static/chunks/3498-3701d2045c9a344c.js"></script><script defer="" src="/_next/static/chunks/5558-29112b2bf574a06d.js"></script><script defer="" src="/_next/static/chunks/5366-90d22618471aab6b.js"></script><script defer="" src="/_next/static/chunks/pages/sports/%5Bid%5D-075d79bc27b0cb0b.js"></script><script defer="" src="/_next/static/pDbetItcMsPxqxInlsybY/_buildManifest.js"></script><script defer="" src="/_next/static/pDbetItcMsPxqxInlsybY/_ssgManifest.js"></script></head><body tabindex="-1"><script async="" crossorigin="anonymous" src="https://kit.fontawesome.com/97ee792e5e.js"></script><div id="__next"><div data-rk=""><style>[data-rk]{--rk-blurs-modalOverlay:blur(0px);--rk-fonts-body:SFRounded, ui-rounded, "SF Pro Rounded", -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol";--rk-radii-actionButton:9999px;--rk-radii-connectButton:12px;--rk-radii-menuButton:12px;--rk-radii-modal:24px;--rk-radii-modalMobile:28px;--rk-colors-accentColor:#3898FF;--rk-colors-accentColorForeground:#FFF;--rk-colors-actionButtonBorder:rgba(255, 255, 255, 0.04);--rk-colors-actionButtonBorderMobile:rgba(255, 255, 255, 0.08);--rk-colors-actionButtonSecondaryBackground:rgba(255, 255, 255, 0.08);--rk-colors-closeButton:rgba(224, 232, 255, 0.6);--rk-colors-closeButtonBackground:rgba(255, 255, 255, 0.08);--rk-colors-connectButtonBackground:#1A1B1F;--rk-colors-connectButtonBackgroundError:#FF494A;--rk-colors-connectButtonInnerBackground:linear-gradient(0deg, rgba(255, 255, 255, 0.075), rgba(255, 255, 255, 0.15));--rk-colors-connectButtonText:#FFF;--rk-colors-connectButtonTextError:#FFF;--rk-colors-connectionIndicator:#30E000;--rk-colors-downloadBottomCardBackground:linear-gradient(126deg, rgba(0, 0, 0, 0) 9.49%, rgba(120, 120, 120, 0.2) 71.04%), #1A1B1F;--rk-colors-downloadTopCardBackground:linear-gradient(126deg, rgba(120, 120, 120, 0.2) 9.49%, rgba(0, 0, 0, 0) 71.04%), #1A1B1F;--rk-colors-error:#FF494A;--rk-colors-generalBorder:rgba(255, 255, 255, 0.08);--rk-colors-generalBorderDim:rgba(255, 255, 255, 0.04);--rk-colors-menuItemBackground:rgba(224, 232, 255, 0.1);--rk-colors-modalBackdrop:rgba(0, 0, 0, 0.5);--rk-colors-modalBackground:#1A1B1F;--rk-colors-modalBorder:rgba(255, 255, 255, 0.08);--rk-colors-modalText:#FFF;--rk-colors-modalTextDim:rgba(224, 232, 255, 0.3);--rk-colors-modalTextSecondary:rgba(255, 255, 255, 0.6);--rk-colors-profileAction:rgba(224, 232, 255, 0.1);--rk-colors-profileActionHover:rgba(224, 232, 255, 0.2);--rk-colors-profileForeground:rgba(224, 232, 255, 0.05);--rk-colors-selectedOptionBorder:rgba(224, 232, 255, 0.1);--rk-colors-standby:#FFD641;--rk-shadows-connectButton:0px 4px 12px rgba(0, 0, 0, 0.1);--rk-shadows-dialog:0px 8px 32px rgba(0, 0, 0, 0.32);--rk-shadows-profileDetailsAction:0px 2px 6px rgba(37, 41, 46, 0.04);--rk-shadows-selectedOption:0px 2px 6px rgba(0, 0, 0, 0.24);--rk-shadows-selectedWallet:0px 2px 6px rgba(0, 0, 0, 0.24);--rk-shadows-walletLogo:0px 2px 16px rgba(0, 0, 0, 0.16);}</style></div></div><script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{}},"page":"/sports/[id]","query":{},"buildId":"pDbetItcMsPxqxInlsybY","nextExport":true,"autoExport":true,"isFallback":false,"locale":"en","locales":["en"],"defaultLocale":"en","scriptLoader":[]}</script></body></html>

可行替代方案 #

PhantomJS已被Selenium官方废弃，推荐使用主流浏览器的无头模式实现JavaScript渲染页面的抓取：

1. Chrome Headless模式 #

Chrome浏览器本身支持无头模式，无需额外安装PhantomJS，代码示例：

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

# 配置Chrome无头选项
chrome_options = Options()
chrome_options.add_argument("--headless=new")
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("user-agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/113.0.5672.127 Safari/537.11")

# 初始化浏览器驱动
driver = webdriver.Chrome(options=chrome_options)

try:
    # 访问目标页面
    driver.get("https://beta.goal3.xyz/sports/soccer")
    
    # 等待页面关键元素加载（替换为实际页面中存在的元素选择器）
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.ID, "__next"))
    )
    
    # 获取渲染后的完整HTML
    rendered_html = driver.page_source
    print(rendered_html)
finally:
    # 关闭浏览器
    driver.quit()

2. Firefox Headless模式 #

Firefox同样支持无头模式，代码示例：

from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

# 配置Firefox无头选项
firefox_options = Options()
firefox_options.add_argument("--headless")
firefox_options.add_argument("user-agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/113.0.5672.127 Safari/537.11")

# 初始化浏览器驱动
driver = webdriver.Firefox(options=firefox_options)

try:
    driver.get("https://beta.goal3.xyz/sports/soccer")
    
    # 等待页面加载完成
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.ID, "__next"))
    )
    
    rendered_html = driver.page_source
    print(rendered_html)
finally:
    driver.quit()

关键说明 #

• 使用WebDriverWait替代sleep，可更精准地等待页面元素加载完成，避免不必要的等待时间。
• 确保已安装对应浏览器的驱动（ChromeDriver或GeckoDriver），并与浏览器版本匹配。

内容的提问来源于stack exchange，提问作者GildedWasp