使用BeautifulSoup爬取Glassdoor职位信息返回None及数据拆分求助

我用下面的代码爬取Glassdoor职位链接列表时，获取id为JobDescriptionContainer的div一直返回None，但页面HTML里明明包含这个div，其下有职位描述、Responsabilities（职责）和Qualifications（任职要求）板块。我需要从900个链接中提取该div下的所有文本，并将职责和任职要求拆分到DataFrame的不同列，求解决办法。

代码示例 #

links = ['https://www.glassdoor.com/partner/jobListing.htm?pos=101&ao=1136043&s=58&guid=0000018209b985b3814e13e6abec3f6b&src=GD_JOB_AD&t=SR&vt=w&ea=1&cs=1_968a1afd&cb=1658020530267&jobListingId=1007823714104&jrtk=3-0-1g84rj1gokf0t801-1g84rj1hdghre800-21320f35b2a9f6f4-', 'https://www.glassdoor.com/partner/jobListing.htm?pos=102&ao=1136043&s=58&guid=0000018209b985b3814e13e6abec3f6b&src=GD_JOB_AD&t=SR&vt=w&ea=1&cs=1_11e4da95&cb=1658020530267&jobListingId=1007830003866&jrtk=3-0-1g84rj1gokf0t801-1g84rj1hdghre800-6ad629ee4ebc1885-', 'https://www.glassdoor.com/partner/jobListing.htm?pos=103&ao=1136043&s=58&guid=0000018209b985b3814e13e6abec3f6b&src=GD_JOB_AD&t=SR&vt=w&cs=1_0ae3fe0c&cb=1658020530267&jobListingId=1008006371431&jrtk=3-0-1g84rj1gokf0t801-1g84rj1hdghre800-f24a3ad703626f08-']

for link in links:
    page = requests.get(link)
    soup = BeautifulSoup(page.text, 'html.parser')
    div = soup.find(id="JobDescriptionContainer")
    print(div)

页面HTML结构示例 #

<div id="Job view">
<div>
    <div>
        <div>
            <span>
            <div>
            <div>
                <header>
                <div>
                    <div>
                        <div id="JobDescriptionContainer">
                            <div>
                                <div>   
                                    <p... text>
                                    <p...text>
                                    <p...text>
                                    <h3>Responsabilities</h3>
                                    <ul>
                                        <li>....</li>
                                        <li>....</li>
                                        <li>....</li>
                                    </ul>
                                    <h3>Qualifications</h3>
                                    <ul>
                                        <li>....</li>
                                        <li>....</li>
                                        <li>....</li>
                                    </ul>

1. 解决JobDescriptionContainer返回None的问题 #

核心原因 #

• Glassdoor有反爬机制，直接用requests.get()获取的是未渲染的静态HTML，目标内容可能由JavaScript动态加载，或者网站检测到非浏览器请求，返回的内容不包含目标div。
• 缺少模拟浏览器的请求头，导致被网站拦截。

具体解决办法 #

方法一：添加请求头模拟浏览器

给requests.get()添加headers参数，模拟正常浏览器请求：

import requests
from bs4 import BeautifulSoup

# 模拟Chrome浏览器请求头
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'
}

for link in links:
    page = requests.get(link, headers=headers)
    soup = BeautifulSoup(page.text, 'html.parser')
    div = soup.find(id="JobDescriptionContainer")
    print(div)

方法二：用Selenium渲染动态页面

如果目标内容是JS动态加载的，用Selenium模拟浏览器打开页面，等待内容加载完成后再提取：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd
import time

# 初始化浏览器（需提前安装对应浏览器的驱动并配置路径）
driver = webdriver.Chrome()
data = []

for link in links:
    try:
        driver.get(link)
        # 等待目标div加载完成，超时时间10秒
        job_desc_container = WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.ID, "JobDescriptionContainer"))
        )
        
        # 提取整个div的文本
        full_text = job_desc_container.text
        
        # 提取职责和任职要求
        responsibilities = []
        qualifications = []
        
        # 遍历所有h3标签，匹配对应板块
        sections = job_desc_container.find_elements(By.TAG_NAME, "h3")
        for section in sections:
            section_text = section.text.strip()
            if "Responsabilities" in section_text:
                # 获取h3后续的第一个ul列表
                ul = section.find_element(By.XPATH, "./following-sibling::ul[1]")
                responsibilities = [li.text.strip() for li in ul.find_elements(By.TAG_NAME, "li")]
            elif "Qualifications" in section_text:
                ul = section.find_element(By.XPATH, "./following-sibling::ul[1]")
                qualifications = [li.text.strip() for li in ul.find_elements(By.TAG_NAME, "li")]
        
        # 存入数据列表
        data.append({
            "full_description": full_text,
            "responsibilities": "\n".join(responsibilities),
            "qualifications": "\n".join(qualifications)
        })
        
        # 添加随机延迟，避免被封
        time.sleep(2 + random.random())
        
    except Exception as e:
        print(f"处理链接 {link} 出错: {str(e)}")
        data.append({
            "full_description": None,
            "responsibilities": None,
            "qualifications": None
        })

driver.quit()
# 转换为DataFrame
df = pd.DataFrame(data)
# 保存到CSV文件
df.to_csv("glassdoor_jobs.csv", index=False)
print(df.head())

2. 批量处理900个链接的注意事项 #

• 添加延迟：每爬取一个链接后，用time.sleep()添加2-5秒的随机延迟，降低被网站封禁IP的风险。
• 异常处理：针对网络超时、元素找不到等情况添加try-except，保证批量任务不会因为单个链接出错而中断。
• 数据备份：定期将DataFrame保存到CSV或Excel文件，防止程序崩溃导致数据丢失。
• IP轮换：如果爬取过程中出现IP被封，可使用代理IP池轮换请求。

内容的提问来源于Stack Exchange，提问作者octaviodiego78