Python从DofusDB API获取JSON数据的验证码及请求问题

DevOps 小满

2026-6-25

问题：自动获取DofusDB寻宝工具的JSON数据失败

背景

我需要从DofusDB的寻宝工具页面获取JSON数据。操作流程为：输入坐标后点击方向箭头，网站会发起4个请求，最后一个请求的响应就是目标数据。

遇到的阻碍

1. Google reCAPTCHA请求处理失败

第一个请求向Google发送带Cookie的数据以获取后续API请求必需的token，但第二个Google相关请求（POST到/recaptcha/enterprise/clr?k=6Lfd6...）无法正常完成，请求示例：

POST /recaptcha/enterprise/clr?k=6Lfd6... HTTP/2
Host: www.google.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0
Accept: /
Accept-Language: en-CA,en-US;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate, br
Content-Type: application/x-protobuf
Content-Length: 1735
Origin: https://www.google.com
Connection: keep-alive
Referer: https://www.google.com/recaptcha/enterprise/anchor?...
Cookie: _GRECAPTCHA=09AKPP-6f4E9WP-7x3Rbln3-n7ZTF2OcVX5Jsfc6XHnBzLMSoYO4X0TO5m98oIWTOe5Phzt9QGeJ3EID9xpzsuAbc
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin

2. API请求无法返回数据

第四个请求是向api.dofusdb.fr发起的GET请求，本该返回JSON，但始终无法获取有效数据，请求示例：

GET /treasure-hunt?x=1&y=2&direction=4&$limit=50&lang=fr HTTP/2
Host: api.dofusdb.fr
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0
Accept: application/json, text/plain, /
Accept-Language: en-CA,en-US;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate, br
token: 03AFcWe...
Origin: https://dofusdb.fr
Connection: keep-alive
Referer: https://dofusdb.fr/
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-site
If-None-Match: W/"137c-U9V1tcXlBJJw8dODoVz3KV3/Fig"
TE: trailers

已尝试的方法

使用requests模块直接请求，四个请求均返回空响应
使用带代理的无头Selenium WebDriver（undetected-chromedriver），页面加载正常，但仍无法获取目标JSON
考虑过改用httpx或hyper，但担心降级Python依赖影响现有代码库

我的Selenium代码

# browser_actions.py

from browsermobproxy import Server
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
import undetected_chromedriver as uc
from selenium.webdriver.common.keys import Keys
from config import TEMPS_SCREENSHOT_PATH, PROXY_PATH
import shutil
import time
import os
import ssl
import json

def start_chrome_driver():
    # Create a SSL context that ignores certificate errors
    ssl_context = ssl.create_default_context()
    ssl_context.check_hostname = False
    ssl_context.verify_mode = ssl.CERT_NONE

    # Start the BrowserMob Proxy server and create a proxy
    server = Server(PROXY_PATH)
    server.start()
    proxy = server.create_proxy()

    #Starts a headless Chrome browser.
    user_agent ="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0"
    options = webdriver.ChromeOptions()
    options.add_argument(f'--proxy-server={proxy.proxy}')
    options.add_argument(f'--user-agent={user_agent}')
    options.add_argument("--headless")
    options.add_argument('--disable-blink-features=AutomationControlled')
    options.add_argument('--ignore-certificate-errors')

    options.add_argument(f'--proxy-server={proxy.proxy}')

    driver = uc.Chrome(options=options)


    return driver, server, proxy

def navigate_to_page(driver, url):
    """Navigates to the specified URL."""
    driver.get(url)
    driver.set_window_size(1920, 1080)

def enter_coordinates(driver, x, y):
    """Enters the coordinates (X, Y) into the input fields on the webpage."""
    input_element = driver.find_element(By.XPATH, '//input[@placeholder="X"]')
    input_element.send_keys(x)
    input_element = driver.find_element(By.XPATH, '//input[@placeholder="Y"]')
    input_element.send_keys(y)

def click_arrow_button(driver, way):
    """Clicks the arrow button on the webpage."""
    left_arrow_button = driver.find_element(By.CLASS_NAME, 'fa-arrow-' + way)
    left_arrow_button.click()


def search_item(driver, search_query):
    try:
        """Enters a search query into the search bar and clicks the search button."""
        search_bar = driver.find_element(By.CSS_SELECTOR, 'input[type="search"]')
        ActionChains(driver).click(search_bar).perform()
        search_bar.send_keys(search_query)
        time.sleep(3)  # Give time for search results to load
        element = driver.find_element(By.XPATH, f'//*[text()="{search_query}"]')
        element.click()
        element.send_keys(Keys.RETURN)
        time.sleep(2)
        
    except:
        driver.save_screenshot(TEMPS_SCREENSHOT_PATH + "\\error.png")

def save_screenshot(driver):
    """Saves a screenshot of the webpage."""
    driver.save_screenshot(TEMPS_SCREENSHOT_PATH + "\\full_page_coords.png")

def delete_screen():
    shutil.rmtree(TEMPS_SCREENSHOT_PATH)
    os.mkdir(TEMPS_SCREENSHOT_PATH)

def suppress_exception_in_del(uc):
    old_del = uc.Chrome.__del__

    def new_del(self) -> None:
        try:
            old_del(self)
        except:
            pass
    
    setattr(uc.Chrome, '__del__', new_del)

def execute_chassebot(url, x, y, way, search_query):
    start_time = time.time()
    """Executes the chassebot actions in the desired order."""
    suppress_exception_in_del(uc)
    delete_screen()
    driver, server, proxy = start_chrome_driver()
    navigate_to_page(driver, url)
    enter_coordinates(driver, x, y)
    proxy.new_har("chassebot")
    click_arrow_button(driver, way)
    search_item(driver, search_query)
    time.sleep(2)
    save_screenshot(driver)
    har_data = proxy.har
    with open("chassebot.har", "w") as har_file:
        json.dump(har_data, har_file)
    driver.quit()
    server.stop()
    end_time = time.time()
    print(f"navigate_to_page took {end_time - start_time} seconds to run")
    time.sleep(1)

execute_chassebot("https://dofusdb.fr/fr/tools/treasure-hunt", 10, 5, "left", "")

当前状态

截图显示页面下拉菜单已加载所有名称，但目标JSON数据仍无法获取，求可行解决方案或指导。

内容的提问来源于stack exchange，提问作者felix st-aubin

可行解决方案建议

修复HAR捕获时机
当前代码在点击箭头后才启动proxy.new_har，会漏掉前三个关键请求。应将HAR捕获移到页面加载后：
```
def execute_chassebot(url, x, y, way, search_query):
    # ... 其他代码
    navigate_to_page(driver, url)
    proxy.new_har("chassebot")  # 移至此处
    enter_coordinates(driver, x, y)
    click_arrow_button(driver, way)
    # ... 其他代码
```
之后检查生成的chassebot.har，查看第四个API请求的响应状态码和内容，确认是请求被拦截还是返回为空。

优化Selenium模拟配置

暂时关闭无头模式（去掉--headless参数），手动完成一次reCAPTCHA验证，验证后复用Cookie和token
添加更多真实用户模拟配置：启用图片加载、添加--start-maximized参数、用随机延迟替代固定time.sleep

改用selenium-wire替代BrowserMob Proxy，它对HTTP/2支持更好：

# 安装：pip install selenium-wire
from seleniumwire import webdriver

def start_chrome_driver():
    options = webdriver.ChromeOptions()
    # 保留原有配置
    driver = uc.Chrome(options=options, seleniumwire_options={'enable_har': True})
    return driver, None, None

然后通过driver.requests筛选目标API请求：

for request in driver.requests:
    if 'api.dofusdb.fr/treasure-hunt' in request.url:
        print(request.response.status_code)
        print(request.response.body.decode('utf-8'))

手动复用Cookie和token验证
手动在浏览器完成一次操作，从开发者工具复制第四个请求的token和Cookie，用requests直接测试：

import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0',
    'Accept': 'application/json, text/plain, */*',
    'token': '复制的token值',
    'Referer': 'https://dofusdb.fr/',
    'Origin': 'https://dofusdb.fr'
}
cookies = {
    # 复制浏览器中的Cookie键值对
}
response = requests.get('https://api.dofusdb.fr/treasure-hunt?x=10&y=5&direction=4&$limit=50&lang=fr', headers=headers, cookies=cookies)
print(response.json())

如果此方法有效，说明问题出在自动获取token和Cookie的环节，可进一步研究如何从浏览器中自动提取这些参数。

Python从DofusDB API获取JSON数据的验证码及请求问题

背景 #

遇到的阻碍 #

1. Google reCAPTCHA请求处理失败 #

2. API请求无法返回数据 #

已尝试的方法 #

我的Selenium代码 #

当前状态 #

可行解决方案建议 #

背景