使用Beautiful Soup爬取FanDuel体育博彩赔率遇内容缺失问题

I’ve been in your exact situation—excited to build a betting strategy with the Kelly Criterion, only to hit a wall when my scraper didn’t pull the odds data I needed. Let’s break down what’s going on and how to fix it:

The Core Problem: Dynamic Content Loading #

FanDuel (like most modern sportsbook platforms) doesn’t embed betting odds directly in the static HTML that requests fetches. Instead, the page loads a basic template first, then uses JavaScript to make background API calls to pull in real-time odds data, which gets rendered on the page afterward.

When you print your BeautifulSoup object, you’re only seeing that initial barebones template—no odds included—because requests can’t execute JavaScript or wait for those dynamic API calls to finish.

How to Fix It: Two Reliable Approaches #

Approach 1: Call FanDuel’s API Directly #

This is the cleaner, faster method if you can track down the right API endpoint. Here’s how to do it:

1. Open your browser’s Developer Tools (F12), go to the Network tab, and refresh the FanDuel page you want to scrape.
2. Filter requests by XHR/Fetch (these are the background API calls). Look for requests returning JSON data with event details and odds—you’ll likely find endpoints like /api/sportsbook/v2/events or similar.
3. Copy that API URL, then use requests to call it directly, mimicking browser headers to avoid being blocked.

Example code for NBA odds:

import requests

# Mimic browser headers to avoid being flagged as a scraper
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36',
    'Referer': 'https://sportsbook.fanduel.com/sports/navigation/830.1/8047.1'
}

# Replace with the actual API endpoint you found in DevTools
api_url = "https://sportsbook.fanduel.com/api/sportsbook/v2/events?sportId=830&leagueId=8047"

response = requests.get(api_url, headers=headers)

if response.status_code == 200:
    odds_data = response.json()
    # Parse the JSON to extract odds (e.g., +102, -120) and feed into your Kelly criterion function
    print(odds_data)
else:
    print(f"API request failed with status code: {response.status_code}")

Approach 2: Use Browser Automation (Selenium) #

If tracking down the API feels too tedious, or if FanDuel’s API requires complex auth/cookies, use a tool that simulates a real browser (which executes JavaScript and loads all dynamic content).

Example with Selenium (headless mode, so no browser window pops up):

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from bs4 import BeautifulSoup

# Configure Chrome to run in headless mode
chrome_options = Options()
chrome_options.add_argument("--headless=new")
chrome_options.add_argument("--disable-gpu")

# Initialize the driver
driver = webdriver.Chrome(options=chrome_options)

# Load the FanDuel NBA page
driver.get('https://sportsbook.fanduel.com/sports/navigation/830.1/8047.1')

# Wait 10 seconds for the page to fully load (adjust as needed)
driver.implicitly_wait(10)

# Get the fully rendered page source
page_source = driver.page_source
soup = BeautifulSoup(page_source, "lxml")

# Search for odds elements (update the class selector to match FanDuel's actual HTML)
odds_elements = soup.find_all(class_='sportsbook-odds')
for elem in odds_elements:
    print(elem.text.strip())

# Clean up
driver.quit()

Key Notes to Keep in Mind #

• API Stability: FanDuel can change their API endpoints at any time, so you may need to re-check the Network tab periodically if your code stops working.
• Terms of Service: Make sure you’re complying with FanDuel’s terms—avoid scraping at high frequencies, which could get your IP blocked.
• Odds Integration: Once you extract the American odds (e.g., +102, -120), you can feed them directly into your existing Kelly criterion function to calculate EV, f, and optimal bets.

内容的提问来源于stack exchange，提问作者rl-pdg