爬取mykhel印度超联球队数据遇403错误及pd.read_html失效求助

问题概述 #

目标抓取页面：https://www.mykhel.com/football/indian-super-league-team-stats-l750/，此前通过pd.read_html可正常解析页面表格，目前出现两个核心问题：

• 直接请求页面返回403 Forbidden错误，被网站反爬机制拦截
• 即使获取到页面源码，pd.read_html返回"No tables found"，页面结构已发生变化

已尝试的失败代码 #

代码1：urllib直接请求 #

from six.moves import urllib
from bs4 import BeautifulSoup
from requests import get
import pandas as pd

url = """https://www.mykhel.com/football/indian-super-league-team-stats-l750/"""

url_contents = urllib.request.urlopen(url).read()

soup = BeautifulSoup(url_contents, "html")
div = soup.find_all("a")

content = pd.DataFrame(div)
content

代码2：Selenium基础配置 #

html5="https://www.mykhel.com/football/indian-super-league-team-stats-l750/"

from selenium import webdriver
from bs4 import BeautifulSoup
import pandas as pd

options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')

site = html5

wd = webdriver.Chrome('chromedriver',options=options)
wd.get(site)
html = wd.page_source
df = pd.read_html(html)
df

代码3：Requests加请求头+BeautifulSoup解析 #

import requests
from selenium import webdriver
from bs4 import BeautifulSoup
import pandas as pd

options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')

test_URL = 'https://www.mykhel.com/football/indian-super-league-team-stats-l750/'

def get_data(link):
    hdr = {'user-agent': 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Mobile Safari/537.36'}

    req = requests.get(link,headers=hdr)
    content = req.content
  
    return content

data = get_data(test_URL)

soup = BeautifulSoup(data, 'html.parser')
Mumplayer = soup.find_all("div",class_= "os-fifastatslist-table")

可行解决方案 #

网站启用了反爬机制，且表格改用div模拟而非标准<table>标签，以下是适配Google Colab的完整爬取代码：

步骤1：安装依赖（Colab环境） #

!pip install selenium chromedriver-autoinstaller

步骤2：爬取代码 #

import pandas as pd
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import chromedriver_autoinstaller

# 自动适配安装对应版本的chromedriver
chromedriver_autoinstaller.install()

# 配置Chrome选项，模拟真实浏览器行为
options = Options()
options.add_argument('--headless=new')  # 新版无头模式，避免被反爬识别
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36')
options.add_argument('--accept-language=en-US,en;q=0.9')
options.add_argument('--referer=https://www.mykhel.com/')

# 初始化浏览器并访问目标页面
driver = webdriver.Chrome(options=options)
driver.get('https://www.mykhel.com/football/indian-super-league-team-stats-l750/')

# 等待表格容器加载完成（最长等待10秒）
wait = WebDriverWait(driver, 10)
table_container = wait.until(EC.presence_of_element_located((By.CLASS_NAME, 'os-fifastatslist-table')))

# 解析div模拟的表格数据
# 提取表头
header_row = table_container.find_elements(By.CLASS_NAME, 'os-fifastatslist-table__head')[0]
headers = [col.text.strip() for col in header_row.find_elements(By.TAG_NAME, 'div')]

# 提取数据行
data_rows = table_container.find_elements(By.CLASS_NAME, 'os-fifastatslist-table__row')
data = []
for row in data_rows:
    cols = row.find_elements(By.TAG_NAME, 'div')
    row_data = [col.text.strip() for col in cols]
    data.append(row_data)

# 转换为DataFrame
df = pd.DataFrame(data, columns=headers)
print(df)

# 关闭浏览器
driver.quit()

关键说明 #

1. 反爬规避：使用新版无头Chrome、完整的用户代理及请求头，模拟真实浏览器访问，解决403问题
2. 动态加载处理：通过WebDriverWait等待表格元素加载，避免因页面未渲染完成导致的元素找不到问题
3. 非标准表格解析：针对网站用div模拟表格的结构，手动提取表头和数据行，确保能正确获取数据

内容的提问来源于stack exchange，提问作者Footilytics - Indian football