设置User-Agent与Session后requests仍返回403，越南房产网站爬取失败

以下是几种可行的解决方案，针对该网站的反爬机制逐一突破：

1. 补充完整请求头字段 #

仅设置User-Agent不足以模拟真实浏览器请求，网站可能校验多个头信息。建议添加Accept、Accept-Language等常见字段：

from bs4 import BeautifulSoup
import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
    'Accept-Language': 'vi-VN,vi;q=0.8,en-US;q=0.5,en;q=0.3',
    'Referer': 'https://batdongsan.com.vn/',
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1'
}

URL = "https://batdongsan.com.vn/ban-can-ho-chung-cu-king-palace"
session = requests.Session()
page = session.get(URL, headers=headers)
print(page.status_code)

2. 使用代理IP轮换 #

如果网站基于IP频率拦截，可通过代理IP切换请求来源（需使用可靠的代理服务）：

from bs4 import BeautifulSoup
import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36'
}

# 替换为实际可用的代理地址
proxies = {
    'http': 'http://your-proxy-ip:port',
    'https': 'https://your-proxy-ip:port'
}

URL = "https://batdongsan.com.vn/ban-can-ho-chung-cu-king-palace"
session = requests.Session()
page = session.get(URL, headers=headers, proxies=proxies)
print(page.status_code)

3. 添加随机请求延迟 #

短时间高频请求会触发反爬，添加随机延迟模拟人类浏览节奏：

from bs4 import BeautifulSoup
import requests
import time
import random

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36'
}

URL = "https://batdongsan.com.vn/ban-can-ho-chung-cu-king-palace"
session = requests.Session()

# 随机延迟2-5秒
time.sleep(random.uniform(2, 5))
page = session.get(URL, headers=headers)
print(page.status_code)

4. 使用自动化浏览器工具 #

若网站依赖JS渲染或有更严格的反爬逻辑，用Selenium模拟真实浏览器操作：

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument('--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36')
options.add_argument('--headless=new')  # 无头模式运行

driver = webdriver.Chrome(options=options)
driver.get("https://batdongsan.com.vn/ban-can-ho-chung-cu-king-palace")

print(driver.page_source)  # 获取完整页面内容
driver.quit()

内容的提问来源于stack exchange，提问作者Phuong Dang