使用Python爬取招聘网站遭遇403错误，求反爬应对方案

针对你爬取Indeed和WeWorkRemotely时遇到的403错误——即使添加User-Agent仍无法解决，可尝试以下几种方案：

1. 补全请求头字段，模拟真实浏览器 #

仅添加User-Agent不足以完全模拟浏览器请求，需要补充更多常见请求头字段，让请求更贴近真实用户的浏览器行为：

headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.5',
    'Referer': 'https://weworkremotely.com/',
    'DNT': '1',
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1'
}

2. 使用会话保持请求状态 #

网站可能通过Cookie验证会话，用requests.Session()自动维持Cookie，模拟浏览器的持续会话状态，避免被识别为爬虫：

# 初始化会话并更新请求头
session = requests.Session()
session.headers.update(headers)

# 先请求首页获取初始Cookie，再请求目标页面
session.get('https://weworkremotely.com/')
time.sleep(2)
response = session.get(url)

3. 添加随机请求延迟 #

频繁请求会触发反爬机制，在请求之间添加随机间隔的延迟，模拟用户浏览节奏：

import random
import time

# 每次请求后添加1-3秒的随机延迟
time.sleep(random.uniform(1, 3))

4. 检查网站robots.txt规则 #

先确认目标网站的/robots.txt文件，查看你要爬取的路径是否被允许，避免违反网站爬取规则。

5. 可选：使用代理IP（以上方法无效时） #

如果网站基于IP限制，可使用可靠代理IP切换请求来源（不建议用公开免费代理，多数质量差且易被拉黑）：

proxies = {
    'http': 'http://your-proxy-ip:port',
    'https': 'https://your-proxy-ip:port'
}

response = session.get(url, proxies=proxies)

修改后的完整代码示例 #

import requests
from bs4 import BeautifulSoup
import pandas as pd
import time
import random

url = 'https://weworkremotely.com/remote-jobs'

headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.5',
    'Referer': 'https://weworkremotely.com/',
    'DNT': '1',
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1'
}

session = requests.Session()
session.headers.update(headers)

# 预热会话获取Cookie
session.get('https://weworkremotely.com/')
time.sleep(random.uniform(1, 3))

response = session.get(url)

if response.status_code == 200:
    soup = BeautifulSoup(response.content, 'html.parser')

    job_titles = []
    companies = []
    locations = []
    job_links = []

    job_sections = soup.find_all('section', class_='jobs')

    for section in job_sections:
        jobs = section.find_all('li', class_='feature')

        for job in jobs:
            title_tag = job.find('span', class_='title')
            title = title_tag.text.strip() if title_tag else 'N/A'
            job_titles.append(title)

            company_tag = job.find('span', class_='company')
            company = company_tag.text.strip() if company_tag else 'N/A'
            companies.append(company)

            location_tag = job.find('span', class_='region company')
            location = location_tag.text.strip() if location_tag else 'Remote'
            locations.append(location)

            job_link_tag = job.find('a', href=True)
            job_link = 'https://weworkremotely.com' + job_link_tag['href'] if job_link_tag else 'N/A'
            job_links.append(job_link)

            # 单个职位解析后添加短延迟
            time.sleep(random.uniform(0.5, 1.5))

    job_data = pd.DataFrame({
        'Job Title': job_titles,
        'Company': companies,
        'Location': locations,
        'Job Link': job_links
    })

    job_data.to_csv('we_work_remotely_jobs.csv', index=False)
    print("Job listings have been successfully saved to we_work_remotely_jobs.csv")
else:
    print(f"Failed to retrieve the webpage. Status code: {response.status_code}")

内容的提问来源于stack exchange，提问作者Yazzine