BeautifulSoup网页爬取问题：无法访问与提取指定元素

Hey there! Congrats on your first Stack Overflow question—welcome to the community! 😊 Let's break down how to solve this scraping task step by step.

1. 定位动态class的ResultsSectionContainer容器 #

Since the container's full class name is dynamic (like ResultsSectionContainer-gdhf14-0 cxyAav), we can target it using a partial class match. BeautifulSoup supports this via CSS selectors, which works perfectly for handling those random suffixes:

# 匹配class包含"ResultsSectionContainer"的div容器
results_container = soup.select_one('div[class*="ResultsSectionContainer"]')

The *= operator here means "class contains this substring"—it ignores the dynamic parts and locks onto the fixed prefix you care about.

2. 遍历所有符合条件的article元素 #

Instead of relying on the dynamic class of <article> tags, we can use their id attribute pattern (since all ids start with job-item-). This is far more reliable! Use the ^= operator in CSS selectors to match ids that start with your target string:

# 从容器中获取所有id以"job-item-"开头的article
job_articles = results_container.select('article[id^="job-item-"]')

3. 提取每个职位的目标信息 #

For each article in the list, we can pull out the exact details you need with simple attribute and text extraction:

job_data = []
for article in job_articles:
    # 提取article的id
    job_id = article.get('id')
    
    # 定位data-at属性为"job-item-title"的a标签
    job_link_tag = article.select_one('a[data-at="job-item-title"]')
    
    # 提取a标签的href属性
    job_href = job_link_tag.get('href')
    
    # 提取h2标签的文本（你说已解决，这里再确认下简洁写法）
    job_title = job_link_tag.find('h2').get_text(strip=True)
    
    # 将数据存入列表统一管理
    job_data.append({
        'job_id': job_id,
        'job_url': job_href,
        'job_title': job_title
    })

# 查看提取结果
for job in job_data:
    print(job)

额外小贴士 #

• If the page loads content dynamically (e.g., via JavaScript after the initial page load), BeautifulSoup alone won't capture it—you'll need tools like selenium to render the full page first.
• Always double-check the website's robots.txt and terms of service to make sure your scraping activities are allowed!

内容的提问来源于stack exchange，提问作者PythonBeginner123