使用Beautiful Soup抓取带加载更多按钮的餐厅标题与描述方法求助

问题 #

使用Beautiful Soup爬取纽约餐厅评论页时，仅能获取21条餐厅标题，无法拿到全部内容——原因是网站通过"加载更多"按钮动态加载数据，静态HTTP请求无法获取未渲染的内容。

现有代码 #

import requests
from bs4 import BeautifulSoup

# 请求网站并下载HTML内容
url='https://www.theinfatuation.com/new-york/reviews'
req=requests.get(url)
content=req.text

soup = BeautifulSoup(content, 'html.parser')
string = soup.find_all("h6")
print(string)

现有输出 #

[<h6 class="styles_headText__4SYjt">All Reviews</h6>, <h6 class="styles_detailedStoryTitle__e3w46">Musette Wine Bar</h6>, <h6 class="styles_detailedStoryTitle__e3w46">Milady's</h6>, <h6 class="styles_detailedStoryTitle__e3w46">Tradisyon</h6>, <h6 class="styles_detailedStoryTitle__e3w46">Lady Wong</h6>, <h6 class="styles_detailedStoryTitle__e3w46">Mamak’s Corner</h6>, <h6 class="styles_detailedStoryTitle__e3w46">Ashes Burnnit</h6>, <h6 class="styles_detailedStoryTitle__e3w46">Padi D'NYC</h6>, <h6 class="styles_detailedStoryTitle__e3w46">Mr. Fried Rice</h6>, <h6 class="styles_detailedStoryTitle__e3w46">Daisy's Dream</h6>, <h6 class="styles_detailedStoryTitle__e3w46">Hainan Jones</h6>, <h6 class="styles_detailedStoryTitle__e3w46">Marufuku Ramen</h6>, <h6 class="styles_detailedStoryTitle__e3w46">Lillie's Victorian Establishment</h6>, <h6 class="styles_detailedStoryTitle__e3w46">Prince Tea House</h6>, <h6 class="styles_detailedStoryTitle__e3w46">The Russian Tea Room</h6>, <h6 class="styles_detailedStoryTitle__e3w46">Maialino (vicino)</h6>, <h6 class="styles_detailedStoryTitle__e3w46">Zaytinya</h6>, <h6 class="styles_detailedStoryTitle__e3w46">Monkey Bar</h6>, <h6 class="styles_detailedStoryTitle__e3w46">Christos Steakhouse</h6>, <h6 class="styles_detailedStoryTitle__e3w46">Peter Luger Steak House</h6>, <h6 class="styles_detailedStoryTitle__e3w46">Buka</h6>]

可行解决方法 #

1. 用Selenium模拟浏览器加载 #

直接模拟点击"加载更多"按钮，等待页面完全渲染后再抓取内容，适合不想分析API的场景。

操作步骤：

• 安装依赖：pip install selenium
• 下载对应浏览器驱动（如ChromeDriver）并配置环境变量
• 编写代码循环点击加载按钮，直到按钮消失

示例代码：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup
import time

# 初始化浏览器
driver = webdriver.Chrome()
driver.get("https://www.theinfatuation.com/new-york/reviews")

# 循环点击加载更多
while True:
    try:
        # 等待加载按钮可点击
        load_btn = WebDriverWait(driver, 10).until(
            EC.element_to_be_clickable((By.CSS_SELECTOR, "button[data-testid='load-more-button']"))
        )
        load_btn.click()
        time.sleep(2)  # 给页面留加载时间
    except:
        # 无更多内容时退出循环
        break

# 解析完整页面HTML
soup = BeautifulSoup(driver.page_source, 'html.parser')
driver.quit()

# 提取餐厅标题和描述
cards = soup.find_all("div", class_="styles_reviewCard__3F8X7")
for card in cards:
    title = card.find("h6", class_="styles_detailedStoryTitle__e3w46").get_text(strip=True)
    desc = card.find("p", class_="styles_summary__2cWp5").get_text(strip=True)
    print(f"标题: {title}\n描述: {desc}\n")

2. 直接调用API接口（高效方案） #

动态加载网站通常通过AJAX请求获取数据，用浏览器开发者工具抓包找到API接口，直接请求接口拿数据，速度更快。

操作步骤：

• 打开浏览器F12开发者工具，切换到Network标签
• 点击页面"加载更多"按钮，观察新出现的XHR请求
• 复制请求URL、Headers和分页参数，编写循环请求代码

示例代码（假设已找到API接口）：

import requests

base_url = "https://api.theinfatuation.com/v2/reviews"
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
    "Referer": "https://www.theinfatuation.com/new-york/reviews"
}
page = 1
all_restaurants = []

while True:
    params = {
        "city_slug": "new-york",
        "page": page,
        "per_page": 20
    }
    resp = requests.get(base_url, headers=headers, params=params)
    if resp.status_code != 200:
        break
    
    data = resp.json()
    if not data["data"]:
        break
    
    all_restaurants.extend(data["data"])
    page += 1

# 提取并打印标题和描述
for item in all_restaurants:
    title = item["attributes"]["name"]
    desc = item["attributes"]["summary"]
    print(f"标题: {title}\n描述: {desc}\n")

注意事项 #

• 不要频繁请求，添加time.sleep(1)类的间隔，避免IP被封禁
• 遵守网站robots.txt规则，不爬取敏感内容
• 遇到反爬时，可更换User-Agent或使用代理IP

内容的提问来源于stack exchange，提问作者TJB