Mac正常运行的BeautifulSoup爬虫在Ubuntu虚拟机遇ConnectionError求助

问题背景 #

• 本地MacBook上基于BeautifulSoup4开发的Python爬虫可正常运行
• 部署到家用Ubuntu虚拟机时触发ConnectionError，已尝试VPN连接、更换多种请求头，问题仍未解决

错误信息 #

{'User-Agent': 'Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7 ChromePlus/1.5.0.0alpha1'}
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.10/http/client.py", line 1374, in getresponse
    response.begin()
  File "/usr/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.10/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

[...]

requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
[Finished in 15.9s with exit code 1]

爬虫代码 #

from bs4 import BeautifulSoup
import requests
import pyuser_agent

URL = f"https://www.edmunds.com/inventory/srp.html?radius=5000&sort=publishDate%3Adesc&pagenumber=2"

ua = pyuser_agent.UA()
headers = {'User-Agent': ua.random}
print(headers)

response = requests.get(url=URL, headers=headers)
soup = BeautifulSoup(response.text, 'lxml')
overview = soup.find()
print(overview)

解决思路 #

• 排查服务器网络连通性
  • 执行ping www.edmunds.com测试基础连通性，traceroute www.edmunds.com检查路由节点是否存在阻断或丢包
  • 对比Mac和Ubuntu的DNS解析结果：在Ubuntu执行nslookup www.edmunds.com，若解析异常可临时切换DNS为8.8.8.8再测试
• 补全请求头字段
  仅替换User-Agent不足以绕过反爬，需添加更多浏览器标准请求头，示例：

  headers = {
      'User-Agent': ua.random,
      'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
      'Accept-Language': 'en-US,en;q=0.5',
      'Referer': 'https://www.edmunds.com/'
  }
  

• 添加请求控制机制
  服务器请求频率易被识别为异常，需加入随机延迟与重试逻辑：

  import time
  import random
  from requests.adapters import HTTPAdapter
  from urllib3.util.retry import Retry
  
  session = requests.Session()
  retry = Retry(total=3, backoff_factor=1, status_forcelist=[429, 500, 502, 503, 504])
  adapter = HTTPAdapter(max_retries=retry)
  session.mount('https://', adapter)
  
  time.sleep(random.uniform(1, 3))  # 随机延迟1-3秒
  response = session.get(url=URL, headers=headers)
  

• 确认VPN实际生效
  在Ubuntu执行curl https://api.ipify.org查看出口IP，与Mac端VPN出口IP对比，确保虚拟机流量确实通过VPN转发
• 验证目标网站反爬策略
  若服务器IP段被目标网站拦截，可尝试使用代理IP；或在Mac上抓取正常请求的Cookie，添加到Ubuntu爬虫的请求头中测试

内容的提问来源于stack exchange，提问作者Irra