爬取JobStreet数据遇'NoneType'无text属性错误，求解决方案

错误原因 #

• 元素定位失败：BeautifulSoup的find()方法找不到目标元素时会返回None，后续直接调用.text或.find()就会触发该错误，比如页面结构变动、部分招聘信息缺失字段。
• 反爬拦截：无请求头的requests.get()容易被识别为爬虫，返回非预期页面内容，导致解析不到目标元素。
• 遍历逻辑缺陷：直接链式调用find()，若外层元素不存在会直接报错，且未处理单个招聘卡片字段缺失的情况。

具体解决方法 #

1. 添加请求头模拟浏览器：给requests.get()传入headers参数，携带User-Agent信息，避免被基础反爬拦截。
2. 逐层校验元素存在性：每次调用find()后先判断结果是否为None，再进行后续操作，避免直接调用属性导致报错。
3. 使用find_all()批量获取元素：用find_all()获取所有招聘卡片，确保遍历的是有效数据集合。
4. 异常捕获机制：对单条数据解析过程添加try-except，即使某条数据解析失败，程序仍能继续处理后续内容。

修正后的完整代码 #

from bs4 import BeautifulSoup as sp
import requests
import openpyxl

# 初始化Excel
excel = openpyxl.Workbook()
sheet = excel.active
sheet.title = 'jobstreet scrap'
sheet.append(['Job', 'link job', 'company', 'location', 'published', 'post'])

keyword = 'sales'
# 添加请求头，模拟浏览器访问
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'
}

for pages in range(1, 10):
    url = f'https://www.jobstreet.co.id/en/job-search/{keyword}-jobs/{pages}'
    try:
        page = requests.get(url, headers=headers)
        page.raise_for_status()  # 检查请求是否成功
        soup = sp(page.content, 'html.parser')
        
        # 先获取外层容器，判断是否存在
        job_container = soup.find('div', class_="sx2jih0 zcydq8bm")
        if not job_container:
            print(f"第{pages}页未找到招聘容器，跳过")
            continue
        
        # 获取所有招聘卡片
        jobs = job_container.find_all('div', class_='sx2jih0')
        if not jobs:
            print(f"第{pages}页无招聘数据，跳过")
            continue
        
        for job in jobs:
            # 解析每个字段，先判断元素是否存在
            job_title_elem = job.find('h1', class_='sx2jih0 zcydq84u _18qlyvc0 _18qlyvc1x _18qlyvc3 _18qlyvca')
            pekerjaan = job_title_elem.a.text.strip() if job_title_elem and job_title_elem.a else 'N/A'
            link_job = job_title_elem.a.get('href') if job_title_elem and job_title_elem.a else 'N/A'
            
            company_elem = job.find('span', class_='sx2jih0 zcydq84u _18qlyvc0 _18qlyvc1x _18qlyvc1 _18qlyvca')
            company = company_elem.text.strip() if company_elem else 'N/A'
            
            location_elem = job.find('span', class_='sx2jih0 zcydq84u _18qlyvc0 _18qlyvc1x _18qlyvc3 _18qlyvc7')
            location = location_elem.span.text.strip() if location_elem and location_elem.span else 'N/A'
            
            time_elem = job.find('time', class_='sx2jih0 zcydq84u')
            published = time_elem.get('datetime') if time_elem else 'N/A'
            pub = time_elem.text.strip() if time_elem else 'N/A'
            
            # 写入Excel
            sheet.append([pekerjaan, link_job, company, location, published, pub])
            print(f'===================== 第{pages}页 =====================')
            print(f"Job : {pekerjaan}\nlink Job : {link_job}\ncompany : {company}\nlocation : {location}\npublished : {pub}\npublished at : {published}")
            
    except Exception as e:
        print(f"第{pages}页处理出错: {str(e)}")
        continue

# 保存Excel
excel.save('jobstreetq.xlsx')

内容的提问来源于stack exchange，提问作者Web Inar