Python网页爬虫无法提取目标文本 已尝试bs4/selenium等工具仍未解决

提取方案 #

该站点为Next.js构建的单页应用，目标数据直接内嵌在页面静态资源中，无需浏览器渲染即可快速提取，示例代码如下：

import requests
import json
from bs4 import BeautifulSoup

url = "https://www.kavak.com/br/carros-usados-100550"
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
}

resp = requests.get(url, headers=headers)
soup = BeautifulSoup(resp.text, "lxml")
next_data = json.loads(soup.find("script", id="__NEXT_DATA__").string)
target_value = next_data["props"]["pageProps"]["car"]["version"]
print(target_value)

运行代码可直接输出结果 G120-5。

如果使用Selenium实现，需要添加显式等待确保元素加载完成后再定位，示例代码如下：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.get("https://www.kavak.com/br/carros-usados-100550")
# 显式等待10秒直到目标元素加载
ele = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.XPATH, "//div[text()='Versão']/following-sibling::span"))
)
print(ele.text)
driver.quit()

内容的提问来源于stack exchange，提问作者renan。