如何用Python抓取JS驱动页面？MSN页面标题提取遇阻求助

尝试抓取MSN页面提取标题「Friday Night Lights」，但无法突破JavaScript加载限制。使用Python结合Selenium和BeautifulSoup工具，尝试过WebDriverWait方法，但执行后返回空列表，获取的页面源码是JavaScript执行前的内容，而浏览器检查器中能看到JS执行后的完整HTML包含目标标题。

原代码：

options = webdriver.ChromeOptions()

options.add_argument("disable-infobars")

options.add_argument("start-maximized")

options.add_argument("disable-dev-shm-usage")

options.add_argument("no-sandbox")

#options.add_experimental_option("prefs", {'profile.managed_default_content_settings.javascript': 2})

options.add_experimental_option("excludeSwitches", ["enable-automation"])

options.add_argument("disable-blink-features=AutomationControlled")

driver = webdriver.Chrome(options=options)

driver.get('https://www.msn.com/nl-be/nieuws/other/de-50-beste-netflix-series-volgens-the-new-york-times/ss-BB1kJC5H?rc=1&ocid=winp1taskbar&cvid=c774477be4b04494b3690631644cf5a9&ei=3#image=2')

page = requests.get('https://www.msn.com/nl-be/nieuws/other/de-50-beste-netflix-series-volgens-the-new-york-times/ss-BB1kJC5H?rc=1&ocid=winp1taskbar&cvid=c774477be4b04494b3690631644cf5a9&ei=3#image=2')

element = WebDriverWait(driver, 30).until(EC.presence_of_element_located((By.CSS_SELECTOR, '#ViewsPageId-BB1kJC5H')))

title = soup.find_all(name="span", class_="title")

print(title)

1. 混用requests和Selenium：requests.get()直接请求静态HTML，不会执行JavaScript，后续用BeautifulSoup解析的是这部分静态内容，而非Selenium渲染后的动态页面。
2. 等待条件不准确：等待的#ViewsPageId-BB1kJC5H元素仅保证父容器加载，但不代表目标标题元素已渲染完成。
3. 未获取正确的页面源码：没有从Selenium的driver.page_source提取渲染后的页面内容，导致解析的是旧的静态HTML。

方案1：结合Selenium与BeautifulSoup #

移除无用的requests请求，等待目标元素后提取Selenium渲染的页面源码进行解析：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup

options = webdriver.ChromeOptions()
options.add_argument("disable-infobars")
options.add_argument("start-maximized")
options.add_argument("disable-dev-shm-usage")
options.add_argument("no-sandbox")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_argument("disable-blink-features=AutomationControlled")

driver = webdriver.Chrome(options=options)
try:
    target_url = 'https://www.msn.com/nl-be/nieuws/other/de-50-beste-netflix-series-volgens-the-new-york-times/ss-BB1kJC5H?rc=1&ocid=winp1taskbar&cvid=c774477be4b04494b3690631644cf5a9&ei=3#image=2'
    driver.get(target_url)
    
    # 直接等待目标标题元素出现，确保JS渲染完成
    WebDriverWait(driver, 30).until(
        EC.presence_of_element_located((By.XPATH, "//span[contains(text(), 'Friday Night Lights')]"))
    )
    
    # 获取渲染后的页面源码
    page_source = driver.page_source
    soup = BeautifulSoup(page_source, 'html.parser')
    
    # 提取目标标题
    target_title = soup.find("span", class_="title", string="Friday Night Lights")
    print(target_title.text if target_title else "未找到目标标题")
finally:
    driver.quit()

方案2：直接用Selenium定位元素（更高效） #

若仅需提取单个标题，无需BeautifulSoup，直接用Selenium定位元素：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.ChromeOptions()
options.add_argument("disable-infobars")
options.add_argument("start-maximized")
options.add_argument("disable-dev-shm-usage")
options.add_argument("no-sandbox")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_argument("disable-blink-features=AutomationControlled")

driver = webdriver.Chrome(options=options)
try:
    target_url = 'https://www.msn.com/nl-be/nieuws/other/de-50-beste-netflix-series-volgens-the-new-york-times/ss-BB1kJC5H?rc=1&ocid=winp1taskbar&cvid=c774477be4b04494b3690631644cf5a9&ei=3#image=2'
    driver.get(target_url)
    
    # 等待标题元素可见并获取文本
    title_element = WebDriverWait(driver, 30).until(
        EC.visibility_of_element_located((By.XPATH, "//span[contains(text(), 'Friday Night Lights')]"))
    )
    print(title_element.text)
finally:
    driver.quit()

内容的提问来源于stack exchange，提问作者remote pbm