如何使用Python和BeautifulSoup抓取Indeed上的完整职位描述
代码问题排查与修复
核心错误点
你的代码存在变量引用错误:你已经对职位详情页发起请求得到job_response对象后,错误将列表页的响应内容response.text赋值给了job_data,导致后续解析的始终是初始列表页的内容,自然找不到仅详情页才存在的id="jobDescriptionText"标签,最终全返回N/A。
修复方案
仅需修改一行代码即可:将job_data = response.text替换为job_data = job_response.text
额外优化建议(避免反爬拦截)
Indeed有反爬机制,建议补充以下配置提升抓取成功率:
- 所有
requests.get()请求添加浏览器请求头,伪装正常用户访问 - 每次请求详情页后添加1-2秒延时,避免请求频率过高被封禁IP
修复后完整代码
from bs4 import BeautifulSoup import requests import pandas as pd import time # 配置请求头 headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36' } url = 'https://www.indeed.com/jobs?q=data+engineer&l=United+States' response = requests.get(url, headers=headers) data = response.text soup = BeautifulSoup(data, 'html.parser') jobs = soup.find_all('div',{'class':'jobsearch-SerpJobCard'}) for job in jobs: title = job.find('a',{'class':'jobtitle'}).text.strip() link1 = job.find('a',{'class':'jobtitle'}).get('href') link = 'https://www.indeed.com' + link1 # 请求职位详情页 job_response = requests.get(link, headers=headers) # 修复:使用详情页的响应内容 job_data = job_response.text job_soup = BeautifulSoup(job_data, 'html.parser') job_description_tag = job_soup.find('div',{'id':'jobDescriptionText'}) job_description = job_description_tag.text.strip() if job_description_tag else "N/A" print('Job Title:', title, '\nLink:', link, '\nJob Description:', job_description, '\n---') # 添加延时 time.sleep(1.5)
内容的提问来源于stack exchange,提问作者Theunis Raubenheimer
相关产品推荐
相关产品推荐

