使用Beautiful Soup无法抓取指定class的div标签内容求助

• 排查动态加载问题：这个网站的内容大概率是JavaScript动态渲染的，requests.get()只能获取初始静态HTML，抓不到JS加载的内容。试试用selenium模拟浏览器加载：

  from selenium import webdriver
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  from bs4 import BeautifulSoup
  
  url = 'https://insights.blackcoffer.com/how-will-covid-19-affect-the-world-of-work-2/'
  driver = webdriver.Chrome()  # 需安装ChromeDriver并配置好路径
  driver.get(url)
  
  # 等待目标元素加载完成
  wait = WebDriverWait(driver, 10)
  wait.until(EC.presence_of_element_located((By.CLASS_NAME, "tdb-block-inner")))
  
  # 获取完整页面源码解析
  page_source = driver.page_source
  soup = BeautifulSoup(page_source, 'html.parser')
  article_content = soup.find('div', class_="tdb-block-inner td-fix-index")
  print(article_content.get_text(strip=True))
  
  driver.quit()
  

• 更换解析器尝试：html.parser对复杂HTML支持有限，换成lxml或html5lib试试：
  先安装依赖：

  pip install lxml html5lib
  

  再修改代码：

  import requests
  from bs4 import BeautifulSoup
  
  url = 'https://insights.blackcoffer.com/how-will-covid-19-affect-the-world-of-work-2/'
  headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'}
  response = requests.get(url, headers=headers)
  soup = BeautifulSoup(response.content, 'lxml')
  article_content = soup.find('div', class_="tdb-block-inner td-fix-index")
  print(article_content.get_text(strip=True) if article_content else '未找到目标元素')
  

• 调整class匹配方式：直接用class_可能匹配失败，试试CSS选择器或多class列表匹配：

  # CSS选择器写法
  article_content = soup.select_one('div.tdb-block-inner.td-fix-index')
  # 多class列表写法
  article_content = soup.find('div', {'class': ['tdb-block-inner', 'td-fix-index']})
  

• 添加请求头防拦截：很多网站会拦截无浏览器标识的请求，给requests.get()加上User-Agent（如上面代码所示），避免被识别为爬虫拦截。

内容的提问来源于stack exchange，提问作者Abhishek Kumar Yadav