Python登录后爬取网页数据：输出格式不符预期的问题求助

问题 #

使用Python的requests和BeautifulSoup库爬取需要登录的higheredjobs网站，当前能获取结果但输出格式混乱，包含大量换行符与无关内容。需要修正代码，实现包含职位标题、公司名称、地点、发布时间的结构化DataFrame输出。

原代码输出示例：

df = 
Jobs title
0   \n\nRe-Sort\n\n\r\n\t\t\tResults 1 - 70 of 70\...
1   \n\n\r\n\t\t\t\t\t\t\t\tAssistant Profes...
2   \r\n\t\t\t\t\t\t\t\r\n\t\t\t\t\t\t\t...

期望输出格式：

df = 
     Jobs title                     Company name                   location         Posted
0   Assistant Professor/Associate  University of Southern Indiana  Evansville, IN   09/02/22
    Professor of Engineering, 
    Pott College of Science, 
    Engineering, and Education - F22057F1

修正后的代码 #

import requests
from bs4 import BeautifulSoup
import pandas as pd

cookies = {
    'CFID': '180615757',
    'CFTOKEN': '64089929988eb934-58E2ACC9-AD21-785B-2AFBCE86106B41FE',
    'visid_incap_2388351': '0Vmr7QpDRvmVw8fbXUJFkB5XEWMAAAAAQUIPAAAAAADtlXunU/D8GLU5VofHHier',
    '_ga_6ZQNJ4ELG2': 'GS1.1.1662315508.15.1.1662315668.0.0.0',
    '_ga': 'GA1.2.147261521.1662080801',
    '_gid': 'GA1.2.1149490171.1662080801',
    'reese84': '3:yMGXsdMquwoCj3IoSFRCMg==:Vf20HwL77P8oWYTTKbE0XigwyQE3d2lLQpPVoZYcoL8SJTmLeqAani+7GspfC2BiJYOOytBlkIp9MewLgs/XbkaiLrSvLnMdZ0aT8/M9FvBohByybnJXNl25ya/yfpGhL9oT1HKMZYnKqSR0Sg8+nHTUEO0/YErJgQmfoeYIT4kmE01S8cndGIemtuGjvq1hzB/D9VAQL7S3idutOumBNu84j5FyCdOBClCJTriE+X9j40lj1swIxFlryTmBAtLHnEvN9M57N4LMb13yuSBaCawrv4fnron0JnUvfKpLU0CXTnpcM9hJNGv9Ekb4Ap43CZDPdeLVzEmj+39wCVtXPtMqBNCU6mPVBSeJCRHyRuQjY+y0Sv5w7ME2LXhT8bEGHyE8yeuxddxvoG51STebu+pb0mSp5n+iKotUEn9h+sA=:WH64twwKGqtE4pUorYOeGylONeXRsfG+3Qe3zAfpdrs=',
    '__atuvc': '65%7C35%2C2%7C36',
    'COOKIESTATUS': 'ON',
    'HIDECOOKIEBANNER': 'TRUE',
    'nlbi_2388351': 'jGGxMFazFBqnU+x+okRrFAAAAAC/AJ/k+R2U+vs5Q4LIRTS7',
    'nlbi_2388351_2147483392': 'PUildkEvtiZ9uje3okRrFAAAAABv1NR/7gPLX7Lc/iS5ei8N',
    'incap_ses_989_2388351': 'mWy+Uq7aLX000xomDaO5DfTrFGMAAAAA6XmB42vG5CO6i609/RhyKg==',
    'incap_ses_468_2388351': 'sDNcR2labTHyNXYlUqx+BipAFGMAAAAAImV2A07lGANZGfpvhvPlLg==',
    '__atuvs': '6314ec0cdbe92a78001',
    '_gat_gtag_UA_12825325_1': '1',
}

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.5',
    'Referer': 'https://www.higheredjobs.com/admin/',
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1',
    'Sec-Fetch-Dest': 'document',
    'Sec-Fetch-Mode': 'navigate',
    'Sec-Fetch-Site': 'same-origin',
}

params = {
    'JobCat': '141',
    'CatName': 'Academic Advising',
}

# 获取页面内容
response = requests.get('https://www.higheredjobs.com/admin/search.cfm', params=params, cookies=cookies, headers=headers)
soup = BeautifulSoup(response.text,'html.parser')

# 定位所有职位记录
job_records = soup.find_all('div', class_='row record')

jobs_data = []
for record in job_records:
    # 提取职位标题，保留换行并清理多余空格
    job_title = record.find('div', class_='col-sm-7').get_text(strip=True, separator='\n')
    # 提取公司名称
    company = record.find('div', class_='institution').get_text(strip=True)
    # 提取地点
    location = record.find('div', class_='location').get_text(strip=True)
    # 提取发布时间，去除前缀
    posted = record.find('div', class_='col-sm-5 text-sm-right').get_text(strip=True).replace('Posted ', '')
    
    jobs_data.append({
        'Jobs title': job_title,
        'Company name': company,
        'location': location,
        'Posted': posted
    })

# 构建结构化DataFrame
df = pd.DataFrame(jobs_data)
print(df.head())

代码说明 #

1. 精准定位职位容器：只抓取每个职位对应的row record元素，避免混入排序栏、结果统计等无关内容。
2. 字段提取优化：
   • 职位标题：用get_text(strip=True, separator='\n')保留标题内的换行结构，同时清理多余空格和制表符。
   • 公司、地点：直接定位专属类名的元素，提取纯文本内容。
   • 发布时间：去除文本前缀Posted ，只保留日期信息。
3. 结构化存储：用字典列表收集每个职位的完整信息，最终转换为格式规整的DataFrame。

内容的提问来源于stack exchange，提问作者Mainland