如何在Python 3.0+中获取网站请求Cookie以完成数据爬取？

我需要爬取marketchameleon.com上苹果（AAPL）的IV30隐含波动率数据，数据接口为https://marketchameleon.com/ivCharts/getIV30_histChartData，该接口需登录后访问，且网站采用Akamai反爬保护，请求时需要包含RT、g_state、_okdetect等17项临时Cookie。

我尝试过requests.Session和Selenium两种方式获取Cookie，但都无法拿到全部必要Cookie，导致请求失败。现在需要Python 3.x环境下能获取完整Cookie的有效方案。

已尝试的代码 #

1. requests.Session 方法 #

import requests
session = requests.Session()
url = 'https://marketchameleon.com/ivCharts/getIV30_histChartData'

headers = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Host': 'marketchameleon.com',
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.2 Safari/605.1.15',
    'Accept-Language': 'en-GB,en;q=0.9',
    'Accept-Encoding': 'gzip, deflate, br',
    'Connection': 'keep-alive',
}

response = session.get(url, headers=headers)
cookies = session.cookies.get_dict()
print(cookies)

2. Selenium 方法 #

from selenium import webdriver
import time

options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options)
driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.53 Safari/537.36'})
driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")

def get_cookies():
    with driver:
        driver.get('https://marketchameleon.com/ivCharts/getIV30_histChartData')
        time.sleep(10)
        driver_cookies = driver.get_cookies()
        cookie = {c['name']:c['value'] for c in driver_cookies}
    return cookie

print(get_cookies())

有效解决方案 #

方案1：优化Selenium流程，模拟完整登录与页面交互 #

Akamai的Cookie通常在页面逐步加载过程中生成，直接访问接口会跳过前置验证与Cookie生成逻辑。正确流程需模拟用户完整操作路径：

1. 访问网站首页，触发Akamai初始验证
2. 模拟账号密码登录
3. 跳转到AAPL的IV30数据页面
4. 提取已生成的完整Cookie

优化后代码示例：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

options = webdriver.ChromeOptions()
options.add_argument("--disable-blink-features=AutomationControlled")
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
options.add_argument("user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.2 Safari/605.1.15")

driver = webdriver.Chrome(options=options)
driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")

def get_valid_cookies(username, password):
    try:
        # 访问首页触发Akamai验证
        driver.get("https://marketchameleon.com/")
        time.sleep(5)

        # 点击登录按钮（元素选择器需根据实际页面调整）
        login_btn = WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.LINK_TEXT, "Login"))
        )
        login_btn.click()
        time.sleep(3)

        # 输入账号密码
        username_input = WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.ID, "username"))
        )
        password_input = driver.find_element(By.ID, "password")
        username_input.send_keys(username)
        password_input.send_keys(password)

        # 提交登录
        submit_btn = driver.find_element(By.ID, "submit-login")
        submit_btn.click()
        time.sleep(5)

        # 跳转到AAPL的IV30页面
        driver.get("https://marketchameleon.com/ivCharts/iv30?symbol=AAPL")
        WebDriverWait(driver, 15).until(
            EC.presence_of_element_located((By.CLASS_NAME, "iv-chart-container"))
        )
        time.sleep(3)

        # 提取完整Cookie
        cookies = {c['name']: c['value'] for c in driver.get_cookies()}
        return cookies
    finally:
        driver.quit()

# 替换为你的账号密码
cookies = get_valid_cookies("your_username", "your_password")
print("完整Cookie列表：", cookies)

方案2：使用Playwright替代Selenium #

Playwright对反爬检测的规避能力更强，能自动处理大部分Akamai验证流程，无需手动设置大量反检测参数：

from playwright.sync_api import sync_playwright

def get_playwright_cookies(username, password):
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=False)  # 测试阶段用非无头模式，后续可改为True
        page = browser.new_page(user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.2 Safari/605.1.15")
        
        # 访问首页完成验证
        page.goto("https://marketchameleon.com/")
        page.wait_for_load_state("networkidle")

        # 执行登录流程
        page.click("text=Login")
        page.fill("#username", username)
        page.fill("#password", password)
        page.click("#submit-login")
        page.wait_for_load_state("networkidle")

        # 跳转到AAPL的IV30页面
        page.goto("https://marketchameleon.com/ivCharts/iv30?symbol=AAPL")
        page.wait_for_selector(".iv-chart-container")

        # 提取Cookie
        cookies = page.context.cookies()
        cookie_dict = {c["name"]: c["value"] for c in cookies}
        browser.close()
        return cookie_dict

# 替换为你的账号密码
cookies = get_playwright_cookies("your_username", "your_password")
print("完整Cookie列表：", cookies)

方案3：手动导出Cookie复用（临时方案） #

如果仅需临时爬取，可手动登录网站后导出Cookie，再在requests中使用：

1. 打开Chrome，登录marketchameleon.com并进入AAPL的IV30页面
2. F12打开开发者工具，切换到Application -> Cookies -> https://marketchameleon.com
3. 右键选择「Copy all cookies」，粘贴到代码中转为字典

示例代码：

import requests

url = 'https://marketchameleon.com/ivCharts/getIV30_histChartData'
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.2 Safari/605.1.15',
    'Accept': 'application/json, text/javascript, */*; q=0.01'
}
# 替换为手动导出的Cookie字典
cookies = {
    'RT': 'your_rt_value',
    'g_state': 'your_g_state_value',
    '_okdetect': 'your_okdetect_value',
    # 补充剩余14项Cookie...
}

response = requests.get(url, headers=headers, cookies=cookies, params={'symbol': 'AAPL'})
print(response.json())

内容的提问来源于stack exchange，提问作者NurseNeko