如何抓取JavaScript渲染的网页数据？以NextGenFTL球员数据为例

需要抓取由JavaScript渲染的网页（https://nextgenftl.com/leagues/ftl-main-2022/game-weeks/week-30/players）中的所有球员徽章、价格及价格变动信息。尝试过以下方法但未成功：

• 直接用BeautifulSoup无法获取JS渲染的数据
• 查看网络面板未找到公开API
• 尝试过Selenium但不清楚如何定位目标数据

已尝试的requests-html代码如下：

from requests_html import HTMLSession
from bs4 import BeautifulSoup

# Assign the URL,
# create the HTMLSession object,
# and run the "get" method to retrieve information from the URL
week = 30
url = f'https://nextgenftl.com/leagues/ftl-main-2022/game-weeks/week-{week}/players'
session = HTMLSession()
response = session.get(url)

# Check that the resolution code was 200
# (successfully retrieved info from URL)
res_code = response.status_code
print(res_code)
if res_code == 200:
    response.html.render() # This is the critical line. This render method runs the script tags to turn them into HTML

    # Get the page content
    soup = BeautifulSoup(response.content, 'lxml')
    print(soup.prettify())
    
else:
    print("Could not reach web page!")

方法一：优化requests-html的解析逻辑 #

你的requests-html代码已经完成了页面渲染，但缺少针对目标数据的提取逻辑。可以直接用requests-html自带的选择器（无需BeautifulSoup）定位元素，步骤如下：

1. 渲染页面时可延长等待时间，确保JS完全加载
2. 通过CSS选择器定位每个球员的行元素
3. 遍历每行提取徽章、价格、价格变动信息

示例代码：

from requests_html import HTMLSession

week = 30
url = f'https://nextgenftl.com/leagues/ftl-main-2022/game-weeks/week-{week}/players'
session = HTMLSession()
response = session.get(url)

if response.status_code == 200:
    # 渲染页面，延长等待时间确保JS加载完成
    response.html.render(sleep=2)
    
    # 定位所有球员行（需根据页面实际CSS类调整，可通过浏览器开发者工具查看）
    player_rows = response.html.find('.player-row')
    
    player_data = []
    for row in player_rows:
        # 提取徽章信息（取img的alt属性）
        badge = row.find('.player-badge img', first=True)
        badge_info = badge.attrs.get('alt', '无徽章') if badge else '无徽章'
        
        # 提取当前价格
        price = row.find('.player-price', first=True).text if row.find('.player-price', first=True) else '无价格'
        
        # 提取价格变动
        price_change = row.find('.price-change', first=True).text if row.find('.price-change', first=True) else '无变动'
        
        player_data.append({
            '徽章': badge_info,
            '价格': price,
            '价格变动': price_change
        })
    
    # 输出或保存数据
    for data in player_data:
        print(data)
else:
    print("无法访问网页！")

方法二：使用Selenium稳定抓取 #

如果requests-html不稳定，Selenium是更可靠的JS渲染页面抓取方案，具体步骤：

1. 提前下载对应浏览器的驱动（如ChromeDriver）并配置路径
2. 访问目标页面，显式等待动态元素加载完成
3. 通过CSS选择器或XPath定位目标数据

示例代码：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

week = 30
url = f'https://nextgenftl.com/leagues/ftl-main-2022/game-weeks/week-{week}/players'

# 初始化Chrome浏览器（需确保ChromeDriver路径正确）
driver = webdriver.Chrome()
driver.get(url)

try:
    # 显式等待球员列表加载完成，最长等待10秒
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CSS_SELECTOR, '.player-row'))
    )
    time.sleep(1)  # 额外等待确保动态数据渲染完成
    
    player_rows = driver.find_elements(By.CSS_SELECTOR, '.player-row')
    player_data = []
    
    for row in player_rows:
        # 提取徽章
        badge_imgs = row.find_elements(By.CSS_SELECTOR, '.player-badge img')
        badge_info = badge_imgs[0].get_attribute('alt') if badge_imgs else '无徽章'
        
        # 提取价格
        price_elems = row.find_elements(By.CSS_SELECTOR, '.player-price')
        price = price_elems[0].text if price_elems else '无价格'
        
        # 提取价格变动
        change_elems = row.find_elements(By.CSS_SELECTOR, '.price-change')
        price_change = change_elems[0].text if change_elems else '无变动'
        
        player_data.append({
            '徽章': badge_info,
            '价格': price,
            '价格变动': price_change
        })
    
    # 输出数据
    for data in player_data:
        print(data)
        
finally:
    driver.quit()

补充：查找隐藏API的技巧 #

如果不想用页面渲染方式，可再尝试以下方法找API：

• 在浏览器开发者工具的「Network」面板中，筛选「XHR」或「Fetch」类型请求，刷新页面后查看是否有返回球员数据的接口
• 查看页面的<script>标签，搜索包含player、price等关键词的JSON数据，部分网站会把数据嵌入在JS变量中

内容的提问来源于stack exchange，提问作者Lucas