树莓派4使用requests-html爬取网页时遭遇Exec format error问题

问题描述 #

我用搭载Pimoroni Inky Frame的树莓派4，想在“Tour for Life”赞助活动中展示最新捐赠金额，数据来自网页https://www.tourforlife.nl/de-rollende-rijders。在MacBook上用Python的requests-html库写的爬取代码能正常获取.gift-amount类的金额，但移植到树莓派4后报错。

代码片段 #

import requests
from bs4 import BeautifulSoup
from requests_html import HTMLSession
from PIL import Image, ImageFont, ImageDraw

def get_donation_amount():
    """ Get the donation amount from the Tour for Life website

    Returns:
        float: The donation amount
    """
    url = "https://www.tourforlife.nl/de-rollende-rijders"

    session = HTMLSession()
    response = session.get(url)
    response.html.render()
    gift_amount = response.html.find('.gift-amount')[0].text
    # remove euro sign and convert to float
    gift_amount = float(gift_amount[2:].replace(",", "."))
    return gift_amount

运行错误输出 #

python3 pidonationtracker.py
[INFO] Starting Chromium download.
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 183M/183M [00:17<00:00, 10.4Mb/s]
[INFO] Beginning extraction
[INFO] Chromium extracted to: /home/inky3/.local/share/pyppeteer/local-chromium/1181205
Traceback (most recent call last):
  File "/home/inky3/PythonProjects/pidonationtracker/src/pidonationtracker/pidonationtracker.py", line 59, in <module>
    donation_amount = get_donation_amount()
  File "/home/inky3/PythonProjects/pidonationtracker/src/pidonationtracker/pidonationtracker.py", line 23, in get_donation_amount
    response.html.render()
  File "/home/inky3/PythonProjects/pidonationtracker/.venv/lib/python3.9/site-packages/requests_html.py", line 586, in render
    self.browser = self.session.browser  # Automatically create a event loop and browser
  File "/home/inky3/PythonProjects/pidonationtracker/.venv/lib/python3.9/site-packages/requests_html.py", line 730, in browser
    self._browser = self.loop.run_until_complete(super().browser)
  File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/home/inky3/PythonProjects/pidonationtracker/.venv/lib/python3.9/site-packages/requests_html.py", line 714, in browser
    self._browser = await pyppeteer.launch(ignoreHTTPSErrors=not(self.verify), headless=True, args=self.__browser_args)
  File "/home/inky3/PythonProjects/pidonationtracker/.venv/lib/python3.9/site-packages/pyppeteer/launcher.py", line 307, in launch
    return await Launcher(options, **kwargs).launch()
  File "/home/inky3/PythonProjects/pidonationtracker/.venv/lib/python3.9/site-packages/pyppeteer/launcher.py", line 148, in launch
    self.proc = subprocess.Popen(  # type: ignore
  File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.9/subprocess.py", line 1823, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 8] Exec format error: '/home/inky3/.local/share/pyppeteer/local-chromium/1181205/chrome-linux/chrome'

错误原因是pyppeteer自动下载的Chromium版本不兼容树莓派的ARM芯片。尝试重装Chrome驱动、安装旧版chromium包均遇到依赖冲突，换Selenium也有Chromium相关故障，求可行解决办法。

可行解决办法 #

1. 优先尝试静态爬取（无需浏览器渲染） #

先确认目标金额是否在静态HTML中输出，不需要JS渲染。如果是，直接用requests+BeautifulSoup替代requests-html，避免浏览器依赖：

import requests
from bs4 import BeautifulSoup

def get_donation_amount():
    url = "https://www.tourforlife.nl/de-rollende-rijders"
    # 添加请求头模拟浏览器，避免被拦截
    headers = {
        "User-Agent": "Mozilla/5.0 (X11; Linux armv7l) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
    }
    response = requests.get(url, headers=headers)
    soup = BeautifulSoup(response.text, 'html.parser')
    gift_amount_element = soup.find(class_='gift-amount')
    if not gift_amount_element:
        raise ValueError("Could not find gift amount element")
    gift_amount = gift_amount_element.text.strip()
    # 处理金额格式：去掉欧元符号，替换逗号为小数点
    return float(gift_amount[2:].replace(",", "."))

2. 使用Playwright（ARM兼容更好） #

Playwright对ARM架构支持更完善，会自动下载适配树莓派的Chromium版本：

1. 安装依赖：

pip install playwright
playwright install chromium

2. 替换后的代码：

from playwright.sync_api import sync_playwright

def get_donation_amount():
    url = "https://www.tourforlife.nl/de-rollende-rijders"
    with sync_playwright() as p:
        # 启动无头浏览器，添加适配树莓派的参数
        browser = p.chromium.launch(headless=True, args=["--no-sandbox", "--disable-dev-shm-usage"])
        page = browser.new_page()
        page.goto(url, wait_until="networkidle")
        # 获取金额文本
        gift_amount = page.locator('.gift-amount').text_content().strip()
        browser.close()
        # 格式化金额
        return float(gift_amount[2:].replace(",", "."))

3. 让requests-html使用系统自带的Chromium #

如果坚持用requests-html，可指定它调用树莓派系统预装的兼容Chromium：

1. 先修复系统Chromium安装：

sudo apt purge chromium-browser chromium-codecs-ffmpeg
sudo apt autoremove
sudo apt install chromium-browser chromium-codecs-ffmpeg

2. 找到Chromium执行路径：

which chromium-browser

输出类似/usr/bin/chromium-browser
3. 修改代码指定该路径：

from requests_html import HTMLSession

def get_donation_amount():
    url = "https://www.tourforlife.nl/de-rollende-rijders"
    session = HTMLSession()
    # 添加浏览器启动参数，适配树莓派环境
    session.__browser_args = ["--no-sandbox", "--disable-dev-shm-usage", "--headless=new"]
    response = session.get(url)
    # 指定系统Chromium路径
    response.html.render(executablePath='/usr/bin/chromium-browser')
    gift_amount = response.html.find('.gift-amount')[0].text.strip()
    return float(gift_amount[2:].replace(",", "."))

内容的提问来源于stack exchange，提问作者Marcel-Jan Krijgsman