使用Beautiful Soup爬取Uber Eats时HTML不一致的解决方法咨询

Uber Eats的商品数据是通过JavaScript动态加载的，requests.get()只能获取初始静态HTML，这就是你看到Beautiful Soup返回的源码和开发者工具不一致的原因。要继续用Beautiful Soup解析，核心思路是先获取JavaScript渲染后的完整页面源码，再传给Beautiful Soup处理。以下是几种可行方案：

方案1：使用Selenium #

Selenium可以模拟真实浏览器打开页面，等待JS渲染完成后再获取源码。

代码示例 #

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from bs4 import BeautifulSoup

ubereats_url = 'https://www.ubereats.com/gb/store/londis-cleadon/nvDm7CwVTBeNK2MzAK6sOQ?diningMode=DELIVERY&pl=JTdCJTIyYWRkcmVzcyUyMiUzQSUyMk5FMzQlMjA4QVElMjIlMkMlMjJyZWZlcmVuY2UlMjIlM0ElMjJDaElKOFY2cDRrOXZma2dSZGNTVmlGTDdhZm8lMjIlMkMlMjJyZWZlcmVuY2VUeXBlJTIyJTNBJTIyZ29vZ2xlX3BsYWNlcyUyMiUyQyUyMmxhdGl0dWRlJTIyJTNBNTQuOTYzNDA4MSUyQyUyMmxvbmdpdHVkZSUyMiUzQS0xLjQxMzMyMDElN0Q%3D'

# 初始化Chrome浏览器（需提前下载对应版本的ChromeDriver并配置路径）
driver = webdriver.Chrome()
driver.get(ubereats_url)

# 等待商品元素加载完成，超时时间10秒
try:
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CLASS_NAME, 'cg'))
    )
finally:
    # 获取渲染后的完整页面源码
    rendered_html = driver.page_source
    driver.quit()

# 用Beautiful Soup解析渲染后的HTML
soup = BeautifulSoup(rendered_html, "lxml")
for product in soup.find_all('div', class_='cg d1 ci d2 ax'):
    print(product.text.strip())

方案2：使用Playwright #

Playwright是更轻量的自动化工具，无需手动下载浏览器驱动，支持无界面运行。

代码示例 #

from playwright.sync_api import sync_playwright
from bs4 import BeautifulSoup

ubereats_url = 'https://www.ubereats.com/gb/store/londis-cleadon/nvDm7CwVTBeNK2MzAK6sOQ?diningMode=DELIVERY&pl=JTdCJTIyYWRkcmVzcyUyMiUzQSUyMk5FMzQlMjA4QVElMjIlMkMlMjJyZWZlcmVuY2UlMjIlM0ElMjJDaElKOFY2cDRrOXZma2dSZGNTVmlGTDdhZm8lMjIlMkMlMjJyZWZlcmVuY2VUeXBlJTIyJTNBJTIyZ29vZ2xlX3BsYWNlcyUyMiUyQyUyMmxhdGl0dWRlJTIyJTNBNTQuOTYzNDA4MSUyQyUyMmxvbmdpdHVkZSUyMiUzQS0xLjQxMzMyMDElN0Q%3D'

with sync_playwright() as p:
    # 启动无界面Chrome浏览器
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto(ubereats_url)
    # 等待商品元素加载完成
    page.wait_for_selector('.cg')
    # 获取渲染后的页面源码
    rendered_html = page.content()
    browser.close()

# 解析HTML并提取商品信息
soup = BeautifulSoup(rendered_html, "lxml")
for product in soup.find_all('div', class_='cg d1 ci d2 ax'):
    print(product.text.strip())

注意：首次使用Playwright需执行playwright install chromium安装浏览器。

方案3：使用Requests-HTML #

Requests-HTML是requests的扩展，内置JS渲染功能，用法更简洁。

代码示例 #

from requests_html import HTMLSession
from bs4 import BeautifulSoup

ubereats_url = 'https://www.ubereats.com/gb/store/londis-cleadon/nvDm7CwVTBeNK2MzAK6sOQ?diningMode=DELIVERY&pl=JTdCJTIyYWRkcmVzcyUyMiUzQSUyMk5FMzQlMjA4QVElMjIlMkMlMjJyZWZlcmVuY2UlMjIlM0ElMjJDaElKOFY2cDRrOXZma2dSZGNTVmlGTDdhZm8lMjIlMkMlMjJyZWZlcmVuY2VUeXBlJTIyJTNBJTIyZ29vZ2xlX3BsYWNlcyUyMiUyQyUyMmxhdGl0dWRlJTIyJTNBNTQuOTYzNDA4MSUyQyUyMmxvbmdpdHVkZSUyMiUzQS0xLjQxMzMyMDElN0Q%3D'

session = HTMLSession()
r = session.get(ubereats_url)
# 执行页面JavaScript，渲染动态内容
r.html.render()

# 获取渲染后的完整HTML
rendered_html = r.html.html

# 用Beautiful Soup解析
soup = BeautifulSoup(rendered_html, "lxml")
for product in soup.find_all('div', class_='cg d1 ci d2 ax'):
    print(product.text.strip())

注意事项 #

• Uber Eats有反爬机制，频繁请求可能触发IP封禁，建议添加请求间隔、使用代理IP。
• 页面元素的class名称可能会动态更新，需定期检查开发者工具中的元素属性是否变化。
• 所有方案都需要安装对应依赖：
  • Selenium：pip install selenium
  • Playwright：pip install playwright
  • Requests-HTML：pip install requests-html

内容的提问来源于stack exchange，提问作者Uni 13