爬虫遍历44页后生成空CSV文件，请求代码问题排查

以下是你的代码中导致CSV为空的核心问题及修复方案：

1. 解析函数逻辑错误 #

parse_job_post_div函数存在两处关键问题：

• 重复查找相同class的元素：先通过soup.findAll("div",{"class":"information"})拿到所有职位块，循环里又用job.find("div",{"class":"information"})，会导致查找失败；同时job_listing错误赋值为BeautifulSoup对象而非文本内容，无法存入CSV。
• 未指定HTML解析器：bs4.BeautifulSoup(div_html)未指定解析器，可能引发解析异常。

修复后的解析函数：

def parse_job_post_div(div_html):
    soup = bs4.BeautifulSoup(div_html, features="html.parser")
    job_ls = soup.findAll("div",{"class":"information"})
    job_data = []
    for job in job_ls:
        job_listing = job.get_text(separator=", ").strip()
        title = job.find("span",{"role":"heading"}).get_text(separator=", ").strip()
        job_location = job.find("p",{"class":"job-info"}).get_text(separator=", ").strip()
        new_row = {"job_listing":job_listing,"title":title,"job_location":job_location}
        job_data.append(new_row)
    
    return job_data

2. 页面元素获取方式错误 #

get_data函数中使用driver.find_element(By.CLASS_NAME, "information")仅能获取第一个职位块，而非页面所有职位，导致解析数据量不足甚至为空。应改为find_elements获取所有元素后逐个处理：

修复后的get_data函数：

def get_data(wd):
    job_postings = driver.find_elements(By.CLASS_NAME, "information")
    parsed_data = []
    for post in job_postings:
        html = post.get_attribute("innerHTML")
        parsed = parse_job_post_div(html)
        parsed_data.extend(parsed)
    return pandas.DataFrame(parsed_data)

3. 循环缩进与页面跳转逻辑问题 #

process_page中的break语句未缩进，会触发语法错误直接终止程序；仅靠time.sleep(10)等待页面加载不可靠，应使用显式等待确保元素加载完成：

修复后的process_page函数：

def process_page(url):
    driver.get(url)
    master_data = []
    i = 0
    while i < MAX_PAGE:
        df = get_data(driver)
        master_data.append(df)
        if i == (MAX_PAGE - 1):
            break
        # 显式等待下一页按钮可点击
        next_btn = WebDriverWait(driver, 20).until(
            EC.element_to_be_clickable((By.XPATH, "//span[@class='icon icon-arrow-right']/parent::a"))
        )
        next_btn.click()
        # 等待新页面职位元素加载完成
        WebDriverWait(driver, 20).until(
            EC.presence_of_all_elements_located((By.CLASS_NAME, "information"))
        )
        print(f"已处理第 {i+1} 页")
        i += 1
    
    return pandas.concat(master_data, ignore_index=True)

4. 基础配置错误 #

• 原URL_BASE缺少参数分隔符?，链接无效，正确链接应为：

  URL_BASE = "https://jobs.pssi.com/us/en/search-results?keywords=%22food%20safety%20team%20member%22&s=1"
  

• 驱动管理混乱：导入GeckoDriverManager但未使用，建议改用webdriver_manager自动管理Chrome驱动，避免版本不兼容：

  from seleniumwire import webdriver
  from webdriver_manager.chrome import ChromeDriverManager
  
  driver = webdriver.Chrome(ChromeDriverManager().install())
  

完整修复后的代码 #

import pandas as pd
import requests
import bs4
import time
from seleniumwire import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import datetime

TODAY = datetime.datetime.today().strftime("%Y%m%d")
SAVE_FILENAME = f"/Users/180284/jupyter-1.0.0/pssi_jobs-{TODAY}.csv"

# 使用webdriver_manager自动管理Chrome驱动
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.implicitly_wait(30)

# 修正后的URL，添加参数分隔符?
URL_BASE = "https://jobs.pssi.com/us/en/search-results?keywords=%22food%20safety%20team%22&s=1"

MAX_PAGE = 44

HEADERS = {
    'From': 'myemail'
}

def interceptor(request):
    del request.headers['From']
    request.headers['From'] = HEADERS["From"]

driver.request_interceptor = interceptor

def parse_job_post_div(div_html):
    soup = bs4.BeautifulSoup(div_html, features="html.parser")
    job_ls = soup.findAll("div",{"class":"information"})
    job_data = []
    for job in job_ls:
        job_listing = job.get_text(separator=", ").strip()
        title = job.find("span",{"role":"heading"}).get_text(separator=", ").strip()
        job_location = job.find("p",{"class":"job-info"}).get_text(separator=", ").strip()
        new_row = {"job_listing":job_listing,"title":title,"job_location":job_location}
        job_data.append(new_row)
    
    return job_data
    
def get_data(wd):
    # 获取所有职位块元素
    job_postings = driver.find_elements(By.CLASS_NAME, "information")
    parsed_data = []
    for post in job_postings:
        html = post.get_attribute("innerHTML")
        parsed = parse_job_post_div(html)
        parsed_data.extend(parsed)
    return pd.DataFrame(parsed_data)

def process_page(url):
    driver.get(url)
    master_data = []
    i = 0
    while i < MAX_PAGE:
        df = get_data(driver)
        master_data.append(df)
        if i == (MAX_PAGE - 1):
            break
        # 等待下一页按钮可点击（定位到父级a标签，span本身不可点击）
        next_btn = WebDriverWait(driver, 20).until(
            EC.element_to_be_clickable((By.XPATH, "//span[@class='icon icon-arrow-right']/parent::a"))
        )
        next_btn.click()
        # 等待新页面的职位元素加载完成
        WebDriverWait(driver, 20).until(
            EC.presence_of_all_elements_located((By.CLASS_NAME, "information"))
        )
        print(f"已处理第 {i+1} 页")
        i += 1
    
    return pd.concat(master_data, ignore_index=True)
    
data = process_page(URL_BASE)
data.to_csv(SAVE_FILENAME, index=False)
driver.quit()

内容的提问来源于stack exchange，提问作者Rachel Gold