使用BeautifulSoup无法爬取网站全部页面的问题排查

尝试爬取Hello Work网站的职位数据，通过计数器拼接URL中的页码实现循环爬取，但每次返回的结果完全一致。以下是相关代码：

导入库与UA配置代码 #

# 爬取Hello Work网站的职位信息

# 导入库
import random
import requests
import csv
from bs4 import BeautifulSoup
from datetime import date

# 配置火狐浏览器的User-Agent
user_agents = [
    "Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Firefox/91.0",
    "Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0",
    "Mozilla/5.0 (X11; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0"
]

random_user_agent = random.choice(user_agents)
headers = {'User-Agent': random_user_agent}

计数器循环逻辑代码 #

i = 0
for i in range(1, 15):  
    url = 'https://www.hellowork.com/fr-fr/emploi/recherche.html?p=' + str(i)
    print(url)
    page = requests.get(url, headers=headers)
    if (page.status_code == 200):
        soup = BeautifulSoup(page.text, 'html.parser')
        jobs = soup.findAll('div', class_=' new action crushed hoverable !tw-p-4 md:!tw-p-6 !tw-rounded-2xl')
    
        # 配置CSV
        csvfile = open('jobList.csv', 'w+', newline='')
        row_list = []  # 存储职位列表
    
        try :
            writer = csv.writer(csvfile)
            writer.writerow(["ID","Job Title","Company Name","Contract type","Location","Publish time","Extract Date"])
            for job in jobs:
                id = job.get('id')
                jobtitle = job.find('h3', class_='!tw-mb-0').a.get_text()
                companyname = job.find('span', class_='tw-mr-2').get_text()
                contracttype = job.find('span', class_='tw-w-max').get_text()
                location = job.find('span', class_='tw-text-ellipsis tw-whitespace-nowrap tw-block tw-overflow-hidden 2xsOld:tw-max-w-[20ch]').get_text()
                publishtime = job.find('span', class_='md:tw-mt-0 tw-text-xsOld').get_text()
                extractdate = date.today()

            row_list = [[id, jobtitle, companyname, contracttype, location, publishtime, extractdate]]
            writer.writerows(row_list)
    finally:
        csvfile.close()

1. 文件写入模式与缩进错误 #

• 问题：每次循环用w+打开CSV会清空原有内容，最终仅保留最后一页数据；同时代码缩进错误，导致每个页面只写入最后一条职位信息。
• 修复：将文件初始化写入表头的操作移到循环外，循环内用a模式追加数据，同时修正缩进确保每条职位都被写入：

# 先写入表头
csvfile = open('jobList.csv', 'w', newline='')
writer = csv.writer(csvfile)
writer.writerow(["ID","Job Title","Company Name","Contract type","Location","Publish time","Extract Date"])
csvfile.close()

i = 0
for i in range(1, 15):  
    url = 'https://www.hellowork.com/fr-fr/emploi/recherche.html?p=' + str(i)
    print(url)
    page = requests.get(url, headers=headers)
    if page.status_code == 200:
        soup = BeautifulSoup(page.text, 'html.parser')
        jobs = soup.findAll('div', class_=' new action crushed hoverable !tw-p-4 md:!tw-p-6 !tw-rounded-2xl')
    
        # 追加模式打开文件
        csvfile = open('jobList.csv', 'a', newline='')
        writer = csv.writer(csvfile)
        
        for job in jobs:
            id = job.get('id')
            jobtitle = job.find('h3', class_='!tw-mb-0').a.get_text()
            companyname = job.find('span', class_='tw-mr-2').get_text()
            contracttype = job.find('span', class_='tw-w-max').get_text()
            location = job.find('span', class_='tw-text-ellipsis tw-whitespace-nowrap tw-block tw-overflow-hidden 2xsOld:tw-max-w-[20ch]').get_text()
            publishtime = job.find('span', class_='md:tw-mt-0 tw-text-xsOld').get_text()
            extractdate = date.today()

            writer.writerow([id, jobtitle, companyname, contracttype, location, publishtime, extractdate])
        csvfile.close()

2. User-Agent未循环更新 #

• 问题：仅初始化时随机选择一次UA，所有请求共用同一个UA，容易被反爬机制识别，导致返回相同页面。
• 修复：将UA随机选择逻辑移到循环内部，每次请求使用不同UA：

i = 0
for i in range(1, 15):  
    # 每次请求前重新随机选择UA
    random_user_agent = random.choice(user_agents)
    headers = {'User-Agent': random_user_agent}
    
    url = 'https://www.hellowork.com/fr-fr/emploi/recherche.html?p=' + str(i)
    print(url)
    page = requests.get(url, headers=headers)
    # 后续代码不变...

3. 反爬机制额外应对建议 #

• 使用requests.Session()维持会话，先请求首页获取Cookie后再发起分页请求；
• 给每个请求添加1-3秒的随机延迟（需导入time库），模拟人类浏览节奏；
• 通过浏览器开发者工具确认分页参数是否正确，部分网站可能不使用p作为页码参数。

内容的提问来源于stack exchange，提问作者K_mns