无头模式下Selenium爬取CoinGecko返回空DataFrame问题排查

在无头模式下使用Selenium爬取CoinGecko网站比特币页面的市场数据时，脚本返回空DataFrame。即使添加了固定等待时间，仍无法检测到表格行。请求日志显示请求正常发送，但未找到目标元素。

代码示例 #

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
import pandas as pd
import time

# Path to your ChromeDriver
chrome_driver_path = 'C:\\Users\\hamid\\OneDrive\\Desktop\\chromedriver-win64\\chromedriver.exe'

# Set up headless mode
options = Options()
options.headless = True
options.add_argument("--window-size=1920,1080")

# Set up the WebDriver
driver = webdriver.Chrome(executable_path=chrome_driver_path, options=options)

# Navigate to the CoinGecko Bitcoin page
driver.get('https://www.coingecko.com/en/coins/bitcoin')

# Wait for the page to load
time.sleep(5)

# Extract data from the page
rows = driver.find_elements(By.XPATH, '//table[@class="table"]/tbody/tr')
market_data = []

for row in rows:
    exchange = row.find_element(By.XPATH, './/td[2]/a').text
    pair = row.find_element(By.XPATH, './/td[3]/a/b').text
    price = row.find_element(By.XPATH, './/td[4]/span').text
    volume_24h = row.find_element(By.XPATH, './/td[5]/span').text
    volume_percentage = row.find_element(By.XPATH, './/td[6]').text
    category = row.find_element(By.XPATH, './/td[7]').text
    updated = row.find_element(By.XPATH, './/td[8]').text

    market_data.append({
        'exchange': exchange,
        'pair': pair,
        'price': price,
        'volume_24h': volume_24h,
        'volume_percentage': volume_percentage,
        'category': category,
        'updated': updated
    })

# Close the WebDriver
driver.quit()

# Convert to DataFrame
df = pd.DataFrame(market_data)
print(df)

运行输出 #

Empty DataFrame
Columns: []
Index: []

核心原因 #

1. 反爬机制识别无头模式：CoinGecko会检测浏览器是否为无头模式，对无头请求返回简化页面或不加载动态数据，导致表格无法被抓取。
2. 静态等待不可靠：time.sleep(5)是固定等待，无法保证页面动态内容（尤其是表格）完全加载完成，可能在等待结束时表格还未渲染。
3. XPath选择器失效：页面结构已更新，原XPath无法匹配到目标表格元素。

解决步骤 #

1. 配置无头模式绕过反爬 #

添加更多模拟正常浏览器的参数，让无头Chrome更接近真实浏览器：

options = Options()
options.headless = True
options.add_argument("--window-size=1920,1080")
# 绕过反爬检测的关键参数
options.add_argument("--disable-blink-features=AutomationControlled")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36")

2. 使用显式等待替代静态等待 #

利用Selenium的WebDriverWait等待表格行加载完成，确保元素存在后再进行抓取：

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# 等待表格行加载，最长等待15秒
rows = WebDriverWait(driver, 15).until(
    EC.presence_of_all_elements_located((By.XPATH, '//table[contains(@class, "table-market")]/tbody/tr'))
)

3. 更新XPath选择器 #

CoinGecko的市场表格类名是table-market而非单纯的table，同时部分子元素的结构也有变化，需要修正XPath以匹配正确元素。

修改后的完整代码 #

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd

# Path to your ChromeDriver
chrome_driver_path = 'C:\\Users\\hamid\\OneDrive\\Desktop\\chromedriver-win64\\chromedriver.exe'

# Set up headless mode with anti-bypass settings
options = Options()
options.headless = True
options.add_argument("--window-size=1920,1080")
options.add_argument("--disable-blink-features=AutomationControlled")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36")

# Set up the WebDriver
driver = webdriver.Chrome(executable_path=chrome_driver_path, options=options)

# Navigate to the CoinGecko Bitcoin page
driver.get('https://www.coingecko.com/en/coins/bitcoin')

try:
    # Wait for the market table rows to load
    rows = WebDriverWait(driver, 15).until(
        EC.presence_of_all_elements_located((By.XPATH, '//table[contains(@class, "table-market")]/tbody/tr'))
    )

    market_data = []
    for row in rows:
        # 提取数据时添加异常捕获，避免单个元素缺失导致脚本中断
        try:
            exchange = row.find_element(By.XPATH, './/td[2]/a').text
            pair = row.find_element(By.XPATH, './/td[3]/div/a').text
            price = row.find_element(By.XPATH, './/td[4]/span').text
            volume_24h = row.find_element(By.XPATH, './/td[5]/span').text
            volume_percentage = row.find_element(By.XPATH, './/td[6]').text
            category = row.find_element(By.XPATH, './/td[7]').text
            updated = row.find_element(By.XPATH, './/td[8]').text

            market_data.append({
                'exchange': exchange,
                'pair': pair,
                'price': price,
                'volume_24h': volume_24h,
                'volume_percentage': volume_percentage,
                'category': category,
                'updated': updated
            })
        except Exception as e:
            print(f"跳过异常行: {str(e)}")
            continue

finally:
    # Close the WebDriver regardless of success/failure
    driver.quit()

# Convert to DataFrame
df = pd.DataFrame(market_data)
print(df)

额外说明 #

• 添加了异常捕获，避免单个行的数据缺失导致整个脚本崩溃。
• 更新了pair元素的XPath，适配页面结构变化。

内容的提问来源于stack exchange，提问作者HamidBee