如何用Python从Flexbox元素抓取数据？Scrapy XPath返回空列表

你的问题核心是目标页面数据由JavaScript动态渲染，requests.get()只能获取静态HTML源码，而你要的fmw-value元素内容是JS加载后才生成的，所以静态HTML里没有对应内容，导致XPath返回空。

下面提供两种可行的解决方案：

方案一：用Scrapy+Playwright渲染动态页面 #

Playwright可以模拟浏览器加载JS内容，是Scrapy处理动态页面的常用工具：

1. 安装依赖

pip install scrapy-playwright

2. 配置Scrapy项目的settings.py

DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
PLAYWRIGHT_LAUNCH_OPTIONS = {
    "headless": True,
    "timeout": 30000,
}

3. 编写爬虫代码

import scrapy
from scrapy_playwright.page import PageCoroutine

class ValuteSpider(scrapy.Spider):
    name = "valute"
    start_urls = ["https://mercati.ilsole24ore.com/tassi-e-valute/valute/contro-euro/cambio/EURUS.FX"]

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url,
                meta={
                    "playwright": True,
                    "playwright_page_coroutines": [
                        PageCoroutine("wait_for_selector", "span.fmw-value"),  # 等待目标元素加载完成
                    ],
                },
            )

    def parse(self, response):
        value = response.xpath('//div[@class="col-3"]//span[@class="fmw-value"]/text()').get()
        self.logger.info(f"抓取到的汇率值: {value}")
        yield {"汇率值": value}

方案二：直接请求数据接口（更高效） #

打开浏览器开发者工具（F12），切换到Network -> XHR标签，刷新页面后可以找到返回汇率数据的API接口。直接请求接口获取JSON数据，无需解析HTML：

import requests

api_url = "https://mercati.ilsole24ore.com/api/v1/quotes/EURUS.FX"
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"
}

response = requests.get(api_url, headers=headers)
data = response.json()
exchange_rate = data["last"]  # 根据实际返回的JSON结构提取字段
print(f"当前汇率: {exchange_rate}")

注意：请求接口时需要带上合适的User-Agent，避免被网站拦截。

内容的提问来源于stack exchange，提问作者Salmar