使用BeautifulSoup、Selenium爬取iframe动态比分报错问题求助

我正尝试从动态网站爬取实时比分，目标站点地址为：https://www.sportingindex.com/spread-betting/cricket/international-odi-matches/group_c.37afeb89-7976-47fd-9f17-29c5777e43cd/2nd-odi-bangladesh-v-sri-lanka。

我知晓需要先获取iframe src中的URL，原本编写了如下代码，但无法正常运行，报错信息为：

"TypeError: 'NoneType' object is not iterable"

请问我遗漏了什么内容？

import requests
from bs4 import BeautifulSoup

s = requests.Session()
r = s.get("https://www.sportingindex.com/spread-betting/cricket/international-odi-matches/group_c.37afeb89-7976-47fd-9f17-29c5777e43cd/2nd-odi-bangladesh-v-sri-lanka")

soup = BeautifulSoup(r.content, "html.parser")
iframe_src = soup.find("iframe").attrs["src"]

r = s.get(f"https:{iframe_src}")
soup = BeautifulSoup(r.content, "html.parser")
for container in soup.find("sb-ms-score"):
    score = container.find("span", {"data-gp-sb-callback": "updateMainScore"}).text.strip()
    print(score)

编辑补充 #

我发现iframe内容是通过JavaScript渲染的，因此改用Selenium编写了更新后的代码，如下：

import requests
from bs4 import BeautifulSoup
from selenium import webdriver

firefox_driver_path = 'geckodriver.exe'
driver = webdriver.Firefox(executable_path=firefox_driver_path)
s = requests.Session()
r = s.get("https://www.sportingindex.com/spread-betting/tennis/utr-pro-match-series-womens/group_b.87d1e4ea-be95-4bc4-99d4-47f105484ecd/r-kolzer-v-b-passola")

soup = BeautifulSoup(r.content, "html.parser")

score_section = soup.find('section', class_='scoreboard gp')

iframe_src = score_section.find("iframe").attrs["src"]

r2 = driver.get(iframe_src)

soup2 = BeautifulSoup(r2.content, "html.parser")

pp = soup2.find('div', class_='sb-ms-inplay')

print(pp)

运行到soup2 = BeautifulSoup(r2.content, "html.parser")这一行时抛出如下错误，我打印iframe_src可正常返回正确的iframe地址，该现象十分奇怪：

Traceback (most recent call last):
  File "C:/SportingIndex v9.py", line 18, in <module>
    soup2 = BeautifulSoup(r2.content, "html.parser")
AttributeError: 'NoneType' object has no attribute 'content'

注：代码中的URL已更换，因为此前的比赛已不再处于直播状态。请问有人能定位到问题所在吗？

问题定位与修复方案 #

1. 第一个版本代码报错原因：
   requests只能获取静态页面源码，iframe内部的动态内容是JS渲染生成的，直接用requests请求iframe地址返回的源码里根本不存在sb-ms-score标签，soup.find("sb-ms-score")返回None，遍历None自然触发非可迭代对象报错。

2. 第二个Selenium版本报错原因：
   Selenium的driver.get()方法没有返回值，执行后直接把页面加载到浏览器实例里，赋值给r2的自然是None，调用r2.content肯定触发属性不存在报错。需要直接从driver对象获取页面源码，用driver.page_source代替r2.content即可。

另外还有几个隐含问题需要调整：

• 获取iframe的src时最好做非空判断，避免页面结构变化直接报错
• 用Selenium加载iframe页面后最好加显式等待，等目标元素加载完成再解析，避免因为JS渲染延迟拿不到内容
• 如果iframe地址是相对路径，需要补全域名再传给driver.get()

修复后的参考代码：

import requests
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

firefox_driver_path = 'geckodriver.exe'
driver = webdriver.Firefox(executable_path=firefox_driver_path)
s = requests.Session()
# 目标页面地址
target_url = "https://www.sportingindex.com/spread-betting/tennis/utr-pro-match-series-womens/group_b.87d1e4ea-be95-4bc4-99d4-47f105484ecd/r-kolzer-v-b-passola"
r = s.get(target_url)
soup = BeautifulSoup(r.content, "html.parser")
# 非空判断
score_section = soup.find('section', class_='scoreboard gp')
if not score_section:
    print("未找到比分板块")
    driver.quit()
    exit()
iframe_tag = score_section.find("iframe")
if not iframe_tag or not iframe_tag.get("src"):
    print("未找到iframe地址")
    driver.quit()
    exit()
iframe_src = iframe_tag["src"]
# 补全协议
if iframe_src.startswith("//"):
    iframe_src = "https:" + iframe_src
# 加载iframe页面
driver.get(iframe_src)
# 等待目标元素加载，最多等10秒
try:
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CLASS_NAME, "sb-ms-inplay"))
    )
except:
    print("加载比分内容超时")
    driver.quit()
    exit()
# 直接从driver拿页面源码
soup2 = BeautifulSoup(driver.page_source, "html.parser")
pp = soup2.find('div', class_='sb-ms-inplay')
print(pp)
# 用完退出浏览器
driver.quit()

内容的提问来源于stack exchange，提问作者Andy