使用Python+Requests+Beautiful Soup爬取YouTube订阅数返回None求解决

Hey there! Let's troubleshoot why you're getting None when trying to scrape a YouTube channel's subscriber count with Requests and Beautiful Soup. I’ve run into this exact issue before—YouTube’s made direct scraping way trickier lately, so here’s what’s going on and how to fix it:

1. The subscriber count isn’t in the static HTML #

When you send a basic requests.get() call, you’re only getting the raw, unrendered HTML of the page. YouTube loads most dynamic content (like subscriber counts, video views, etc.) using JavaScript after the initial page loads. That means Beautiful Soup can’t find the element you’re targeting because it doesn’t exist in the static response.

2. YouTube’s anti-scraping measures might be blocking you #

Even if you could find the element in static HTML, YouTube actively blocks non-browser requests. The default requests user-agent tells YouTube you’re a script, not a real browser. Try adding a realistic user-agent header to your request:

import requests
from bs4 import BeautifulSoup

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36'
}

channel_url = "https://www.youtube.com/@YourChannel"
response = requests.get(channel_url, headers=headers)
soup = BeautifulSoup(response.text, 'html.parser')

# Try finding the subscriber element (note: this might still fail due to JS rendering)
subscriber_element = soup.find("yt-formatted-string", {"id": "subscriber-count"})
print(subscriber_element.text if subscriber_element else "None")

But fair warning: even with a proper user-agent, this might still return None because the subscriber count isn’t in the static HTML anymore.

3. The reliable solution: Use the YouTube Data API #

If you want consistent results, the official YouTube Data API is the way to go. It’s free for small-scale use, and you won’t have to fight anti-scraping or dynamic content issues. Here’s how to use it:

1. Go to the Google Cloud Console, create a project, enable the YouTube Data API v3, and generate an API key.
2. Use this code to fetch the subscriber count:

import requests

api_key = "YOUR_GOOGLE_API_KEY"
channel_id = "UCXXXXXXXXXXXXXXXXXXXXXX"  # Find this in your channel's URL (e.g., /channel/UC...)

url = f"https://www.googleapis.com/youtube/v3/channels?part=statistics&id={channel_id}&key={api_key}"
response = requests.get(url)
data = response.json()

if data.get("items"):
    subscriber_count = data["items"][0]["statistics"]["subscriberCount"]
    print(f"Subscriber count: {subscriber_count}")
else:
    print("Could not retrieve channel data—check your API key or channel ID.")

4. If you really want to scrape (not recommended) #

If you’re set on scraping instead of using the API, you’ll need a tool that can render JavaScript. Selenium is a popular option—it simulates a real browser, so it loads all dynamic content. Here’s a quick example:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--headless=new")  # Run in background without a window
driver = webdriver.Chrome(options=options)

channel_url = "https://www.youtube.com/@YourChannel"
driver.get(channel_url)

subscriber_element = driver.find_element(By.ID, "subscriber-count")
print(subscriber_element.text)

driver.quit()

Just keep in mind: YouTube can detect Selenium too, and they might block your requests if you scrape too aggressively. Plus, page elements change often, so this code could break without warning.

内容的提问来源于stack exchange，提问作者bobbybeamer