Python爬取batdongsan.com.vn时加User-Agent仍遇403 Error，求解决

仅添加User-Agent通常不足以绕过该网站的反爬机制，它可能还会验证请求头的其他字段、会话状态或请求频率。以下是几个可行的解决思路：

1. 完善请求头字段 #

补充浏览器发送的常见头信息，让请求更接近真实用户的浏览器请求：

import requests

url3 = "https://batdongsan.com.vn/nha-dat-ban-tp-hcm"
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.114 Safari/537.36 Edg/103.0.1264.49",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
    "Accept-Language": "vi-VN,vi;q=0.9,en-US;q=0.8,en;q=0.7",
    "Accept-Encoding": "gzip, deflate, br",
    "Referer": "https://batdongsan.com.vn/",
    "Connection": "keep-alive",
    "Upgrade-Insecure-Requests": "1"
}

page = requests.get(url3, headers=headers)
print(page.status_code)

2. 用Session维持会话状态 #

网站可能通过Cookie验证请求合法性，使用requests.Session()可自动处理会话Cookie，模拟真实浏览器的会话流程：

import requests

url3 = "https://batdongsan.com.vn/nha-dat-ban-tp-hcm"
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.114 Safari/537.36 Edg/103.0.1264.49",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
}

session = requests.Session()
session.headers.update(headers)
# 先访问首页获取初始Cookie
session.get("https://batdongsan.com.vn/")
page = session.get(url3)
print(page.status_code)

3. 添加随机请求延迟 #

频繁请求会触发反爬机制，在请求之间添加随机延迟，模拟人类浏览的节奏：

import requests
import time
import random

url3 = "https://batdongsan.com.vn/nha-dat-ban-tp-hcm"
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.114 Safari/537.36 Edg/103.0.1264.49",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
}

# 随机延迟1-3秒
time.sleep(random.uniform(1, 3))
page = requests.get(url3, headers=headers)
print(page.status_code)

4. 模拟真实浏览器（进阶） #

如果网站使用了JavaScript渲染或更严格的反爬规则，可使用Selenium模拟完整浏览器行为：

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

url3 = "https://batdongsan.com.vn/nha-dat-ban-tp-hcm"

options = Options()
options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.114 Safari/537.36 Edg/103.0.1264.49")
# 无头模式（不显示浏览器窗口）
options.add_argument("--headless=new")

driver = webdriver.Chrome(options=options)
driver.get(url3)
# 打印页面前500字符验证是否成功获取内容
print(driver.page_source[:500])
driver.quit()

注意：使用Selenium需提前安装对应浏览器的驱动（如ChromeDriver），确保驱动版本与浏览器版本匹配。

内容的提问来源于stack exchange，提问作者Duc Huy NGUYEN