使用BeautifulSoup爬取Binance P2P数据时仅获style标签内容的问题

我尝试从Binance平台爬取USDT/RUB的P2P价格数据，通过定位HTML的div和class来实现，代码如下：

import requests
from bs4 import BeautifulSoup
from fake_useragent import UserAgent

user_agent = UserAgent(verify_ssl=False)
headers = {'user-agent': user_agent.random}

url = 'https://p2p.binance.com/ru/trade/all-payments/USDT?fiat=RUB'
res = requests.get(url, headers=headers).text
soup = BeautifulSoup(res, 'lxml')

lst = soup.find('div', class_="css-186r813")
underlst = lst.find('div', class_="css-cjwhpx")

print(underlst)

但运行后返回的并非预期的交易数据div内容，而是包含<style>标签的加载动画HTML，输出内容如下：

<div class="css-cjwhpx"><style data-emotion-css="62wjvl">.css-62wjvl{box-sizing:border-box;margin:0;min-width:0;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;width:100%;padding-top:80px;padding-bottom:80px;}</style><div class="css-62wjvl"><style data-emotion-css="14f6ssc">.css-14f6ssc{box-sizing:border-box;margin:0;min-width:0;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;width:30px;height:30px;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;-webkit-box-pack:justify;-webkit-justify-content:space-between;-ms-flex-pack:justify;justify-content:space-between;width:48px;height:48px;margin:auto;}</style><div class="css-14f6ssc"><style data-emotion-css="2bc2yz animation-hqgv56">.css-2bc2yz{box-sizing:border-box;margin:0;min-width:0;background-color:#F0B90B;height:100%;width:3px;-webkit-animation:animation-hqgv56 1.2s infinite ease-in-out;animation:animation-hqgv56 1.2s infinite ease-in-out;-webkit-animation-delay:-0.3s;animation-delay:-0.3s;}@-webkit-keyframes animation-hqgv56{0%,40%,100%{-webkit-transform:scaleY(0.4);-ms-transform:scaleY(0.4);transform:scaleY(0.4);}20%{-webkit-transform:scaleY(1.0);-ms-transform:scaleY(1.0);transform:scaleY(1.0);}}@keyframes animation-hqgv56{0%,40%,100%{-webkit-transform:scaleY(0.4);-ms-transform:scaleY(0.4);transform:scaleY(0.4);}20%{-webkit-transform:scaleY(1.0);-ms-transform:scaleY(1.0);transform:scaleY(1.0);}}</style><div class="css-2bc2yz"></div><style data-emotion-css="i6pav4 animation-hqgv56">.css-i6pav4{box-sizing:border-box;margin:0;min-width:0;background-color:#F0B90B;height:100%;width:3px;-webkit-animation:animation-hqgv56 1.2s infinite ease-in-out;animation:animation-hqgv56 1.2s infinite ease-in-out;-webkit-animation-delay:-0.2s;animation-delay:-0.2s;}@-webkit-keyframes animation-hqgv56{0%,40%,100%{-webkit-transform:scaleY(0.4);-ms-transform:scaleY(0.4);transform:scaleY(0.4);}20%{-webkit-transform:scaleY(1.0);-ms-transform:scaleY(1.0);transform:scaleY(1.0);}}@keyframes animation-hqgv56{0%,40%,100%{-webkit-transform:scaleY(0.4);-ms-transform:scaleY(0.4);transform:scaleY(0.4);}20%{-webkit-transform:scaleY(1.0);-ms-transform:scaleY(1.0);transform:scaleY(1.0);}}</style><div class="css-i6pav4"></div><style data-emotion-css="o1uf9l animation-hqgv56">.css-o1uf9l{box-sizing:border-box;margin:0;min-width:0;background-color:#F0B90B;height:100%;width:3px;-webkit-animation:animation-hqgv56 1.2s infinite ease-in-out;animation:animation-hqgv56 1.2s infinite ease-in-out;-webkit-animation-delay:-0.1s;animation-delay:-0.1s;}@-webkit-keyframes animation-hqgv56{0%,40%,100%{-webkit-transform:scaleY(0.4);-ms-transform:scaleY(0.4);transform:scaleY(0.4);}20%{-webkit-transform:scaleY(1.0);-ms-transform:scaleY(1.0);transform:scaleY(1.0);}}@keyframes animation-hqgv56{0%,40%,100%{-webkit-transform:scaleY(0.4);-ms-transform:scaleY(0.4);transform:scaleY(0.4);}20%{-webkit-transform:scaleY(1.0);-ms-transform:scaleY(1.0);transform:scaleY(1.0);}}</style><div class="css-o1uf9l"></div><style data-emotion-css="14jhrrk animation-hqgv56">.css-14jhrrk{box-sizing:border-box;margin:0;min-width:0;background-color:#F0B90B;height:100%;width:3px;-webkit-animation:animation-hqgv56 1.2s infinite ease-in-out;animation:animation-hqgv56 1.2s infinite ease-in-out;-webkit-animation-delay:-0s;animation-delay:-0s;}@-webkit-keyframes animation-hqgv56{0%,40%,100%{-webkit-transform:scaleY(0.4);-ms-transform:scaleY(0.4);transform:scaleY(0.4);}20%{-webkit-transform:scaleY(1.0);-ms-transform:scaleY(1.0);transform:scaleY(1.0);}}@keyframes animation-hqgv56{0%,40%,100%{-webkit-transform:scaleY(0.4);-ms-transform:scaleY(0.4);transform:scaleY(0.4);}20%{-webkit-transform:scaleY(1.0);-ms-transform:scaleY(1.0);transform:scaleY(1.0);}}</style><div class="css-14jhrrk"></div></div></div></div>

我期望获取的是浏览器检查HTML时看到的交易数据div内容，请问哪里操作出错了？

原因分析与解决方案 #

核心问题 #

1. 页面动态渲染：Binance P2P页面的交易数据是通过JavaScript动态加载的，你用requests直接请求得到的只是初始静态HTML，里面只有加载动画，实际数据还没被渲染出来。
2. 动态class不可靠：你用的css-186r813、css-cjwhpx这类class是前端框架动态生成的，会随着页面更新随时变化，依赖它们定位元素极不稳定。

方案1：调用官方API（推荐） #

Binance提供了公开的P2P数据API，比爬取页面更稳定，也不会触发反爬，代码示例：

import requests

url = "https://p2p.binance.com/bapi/c2c/v2/friendly/c2c/adv/search"
# 可根据需求调整参数，比如tradeType改为SELL获取卖出价
payload = {
    "proMerchantAds": False,
    "page": 1,
    "rows": 10,
    "payTypes": [],
    "countries": [],
    "publisherType": None,
    "asset": "USDT",
    "fiat": "RUB",
    "tradeType": "BUY"
}
headers = {
    "Content-Type": "application/json",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"
}

response = requests.post(url, json=payload, headers=headers)
data = response.json()

# 提取并打印价格
for item in data.get('data', []):
    price = item['adv']['price']
    print(f"价格: {price} RUB/USDT")

方案2：用Selenium渲染动态页面 #

如果一定要爬取页面，需要用Selenium模拟浏览器加载，等待动态内容渲染完成：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup

# 需提前安装ChromeDriver并配置环境变量
driver = webdriver.Chrome()
url = "https://p2p.binance.com/ru/trade/all-payments/USDT?fiat=RUB"
driver.get(url)

try:
    # 等待交易项加载完成，用data-testid定位（这个属性不会轻易变化）
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CSS_SELECTOR, '[data-testid="adv-item"]'))
    )
    # 获取渲染后的页面源码
    soup = BeautifulSoup(driver.page_source, 'lxml')
    # 提取所有价格元素
    price_elements = soup.select('[data-testid="adv-item"] [data-testid="price"]')
    for elem in price_elements:
        print(elem.text.strip())
finally:
    # 关闭浏览器
    driver.quit()

注意事项 #

• 永远优先使用官方API，避免反爬风险，数据也更准确。
• 爬取页面时，不要用动态生成的class定位元素，优先选择data-testid、id这类固定属性，或者语义化的标签结构。
• 频繁请求可能触发Binance的反爬机制，建议添加合理的请求间隔。

内容的提问来源于stack exchange，提问作者rockzxm