如何爬取HigherEdJobs职位详情页的薪资、URL及截止日期？

要获取职位详情页的URL、薪资和截止日期，你需要在现有代码基础上添加以下步骤：提取详情页链接、请求详情页内容、解析目标字段。以下是修改后的完整代码及说明：

修改后的代码 #

from pprint import pprint
import datetime
import requests
from bs4 import BeautifulSoup
import pandas as pd
import time

cookies = {
    'CFID': '180615757',
    'CFTOKEN': '64089929988eb934-58E2ACC9-AD21-785B-2AFBCE86106B41FE',
    'visid_incap_2388351': '0Vmr7QpDRvmVw8fbXUJFkB5XEWMAAAAAQUIPAAAAAADtlXunU/D8GLU5VofHHier',
    '_ga_6ZQNJ4ELG2': 'GS1.1.1662315508.15.1.1662315668.0.0.0',
    '_ga': 'GA1.2.147261521.1662080801',
    '_gid': 'GA1.2.1149490171.1662080801',
    'reese84': '3:yMGXsdMquwoCj3IoSFRCMg==:Vf20HwL77P8oWYTTKbE0XigwyQE3d2lLQpPVoZYcoL8SJTmLeqAani+7GspfC2BiJYOOytBlkIp9MewLgs/XbkaiLrSvLnMdZ0aT8/M9FvBohByybnJXNl25ya/yfpGhL9oT1HKMZYnKqSR0Sg8+nHTUEO0/YErJgQmfoeYIT4kmE01S8cndGIemtuGjvq1hzB/D9VAQL7S3idutOumBNu84j5FyCdOBClCJTriE+X9j40lj1swIxFlryTmBAtLHnEvN9M57N4LMb13yuSBaCawrv4fnron0JnUvfKpLU0CXTnpcM9hJNGv9Ekb4Ap43CZDPdeLVzEmj+39wCVtXPtMqBNCU6mPVBSeJCRHyRuQjY+y0Sv5w7ME2LXhT8bEGHyE8yeuxddxvoG51STebu+pb0mSp5n+iKotUEn9h+sA=:WH64twwKGqtE4pUorYOeGylONeXRsfG+3Qe3zAfpdrs=',
    '__atuvc': '65%7C35%2C2%7C36',
    'COOKIESTATUS': 'ON',
    'HIDECOOKIEBANNER': 'TRUE',
    'nlbi_2388351': 'jGGxMFazFBqnU+x+okRrFAAAAAC/AJ/k+R2U+vs5Q4LIRTS7',
    'nlbi_2388351_2147483392': 'PUildkEvtiZ9uje3okRrFAAAAABv1NR/7gPLX7Lc/iS5ei8N',
    'incap_ses_989_2388351': 'mWy+Uq7aLX000xomDaO5DfTrFGMAAAAA6XmB42vG5CO6i609/RhyKg==',
    'incap_ses_468_2388351': 'sDNcR2labTHyNXYlUqx+BipAFGMAAAAAImV2A07lGANZGfpvhvPlLg==',
    '__atuvs': '6314ec0cdbe92a78001',
    '_gat_gtag_UA_12825325_1': '1',
}

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.5',
    'Referer': 'https://www.higheredjobs.com/admin/',
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1',
    'Sec-Fetch-Dest': 'document',
    'Sec-Fetch-Mode': 'navigate',
    'Sec-Fetch-Site': 'same-origin',
}

# 使用Session维持会话，自动处理Cookie
session = requests.Session()
session.headers.update(headers)
session.cookies.update(cookies)

params = {
    'JobCat': '141',
    'CatName': 'Academic Advising',
}

response = session.get('https://www.higheredjobs.com/admin/search.cfm', params=params)
soup = BeautifulSoup(response.text,'html.parser')

jobs_list = []
for i in soup.select('.row.record'):
    # 获取基础职位信息
    basic_info = list(i.stripped_strings)
    job_data = dict(zip(['title','university','location','study','date'], basic_info))
    
    # 提取详情页URL
    a_tag = i.find('a', href=True)
    if a_tag:
        relative_url = a_tag['href']
        full_url = 'https://www.higheredjobs.com' + relative_url
        job_data['url'] = full_url
        
        try:
            # 请求详情页
            detail_response = session.get(full_url)
            detail_response.raise_for_status()
            detail_soup = BeautifulSoup(detail_response.text, 'html.parser')
            
            # 提取薪资
            salary = ''
            salary_label = detail_soup.find(text=lambda t: t and 'Salary:' in t)
            if salary_label:
                salary = salary_label.next_sibling.strip()
            job_data['salary'] = salary
            
            # 提取截止日期
            closing_date = ''
            closing_label = detail_soup.find(text=lambda t: t and 'Closing Date:' in t)
            if closing_label:
                closing_date = closing_label.next_sibling.strip()
            job_data['closing date'] = closing_date
            
            # 添加延迟，避免触发反爬机制
            time.sleep(1)
            
        except Exception as e:
            print(f"获取详情页失败 {full_url}: {e}")
            job_data['salary'] = ''
            job_data['closing date'] = ''
    else:
        job_data['url'] = ''
        job_data['salary'] = ''
        job_data['closing date'] = ''
    
    jobs_list.append(job_data)

df = pd.DataFrame(jobs_list)
print(df)

关键修改说明 #

1. 会话维持：使用requests.Session()替代单次请求，自动保持Cookie有效性，避免重复传递Cookie参数。
2. 详情页URL提取：从每个职位记录的<a>标签中获取相对链接，拼接成完整的详情页URL。
3. 详情页解析：
   • 通过文本匹配找到"Salary:"和"Closing Date:"标签，提取其后的内容作为目标字段。
   • 加入异常处理，避免单个详情页请求失败导致整个程序中断。
4. 反爬策略：添加1秒延迟，降低请求频率，减少被网站封禁的风险。

内容的提问来源于stack exchange，提问作者Mainland