如何解决Python requests出现Connection reset by peer错误且curl可正常访问的问题

补充说明：目标站点可以响应命令行curl请求，但无法响应Python requests的请求。
我有丰富的站点爬虫开发经验，曾抓取过大量站点、提取过数百GB数据，但此前从未遇到过此类问题。目前我无法理解站点为何会抛出“104, connection reset by peer”错误，不清楚遗漏了什么配置或参数。

复现代码 #

from requests import Session
link = 'http://103.94.204.78/searchengine/searchen.aspx'

headers = {
           'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
           'Accept-Encoding': 'gzip, deflate',
           'Accept-Language': 'en-GB,en-US;q=0.9,en;q=0.8',
           'Cache-Control': 'max-age=0',
           'Connection': 'keep-alive',
           'DNT': '1',
           'Host': '103.94.204.78',
           'Upgrade-Insecure-Requests': '1',
           'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36 OPR/80.0.4170.72'
}

with Session() as session:
    resp = session.post(url=link, headers=headers)
    print(resp.text)

报错信息 #

Traceback (most recent call last):
  File "/home/naveed/myenv/lib/python3.8/site-packages/urllib3/response.py", line 438, in _error_catcher
    yield
  File "/home/naveed/myenv/lib/python3.8/site-packages/urllib3/response.py", line 519, in read
    data = self._fp.read(amt) if not fp_closed else b""
  File "/usr/lib/python3.8/http/client.py", line 455, in read
    n = self.readinto(b)
  File "/usr/lib/python3.8/http/client.py", line 499, in readinto
    n = self.fp.readinto(b)
  File "/usr/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/naveed/myenv/lib/python3.8/site-packages/requests/models.py", line 758, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/home/naveed/myenv/lib/python3.8/site-packages/urllib3/response.py", line 576, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/home/naveed/myenv/lib/python3.8/site-packages/urllib3/response.py", line 541, in read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/naveed/myenv/lib/python3.8/site-packages/urllib3/response.py", line 455, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./preproc.py", line 36, in <module>
    resp = session.get(url=link, headers=headers)
  File "/home/naveed/myenv/lib/python3.8/site-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/home/naveed/myenv/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/naveed/myenv/lib/python3.8/site-packages/requests/sessions.py", line 697, in send
    r.content
  File "/home/naveed/myenv/lib/python3.8/site-packages/requests/models.py", line 836, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "/home/naveed/myenv/lib/python3.8/site-packages/requests/models.py", line 761, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

当前环境已经安装了pyopenssl ndg-httpsclient pyasn1依赖包，requests版本为2.26.0。

• 优先模拟curl的指纹特征：curl能通说明站点拦截的是requests的特征，而非IP或者请求参数。requests默认的TLS JA3指纹、请求头排序规则和curl差异很大，这类特征被WAF识别后会直接重置连接。可以直接使用curl_cffi库替换requests，该库可以1:1复刻curl的TLS和HTTP2指纹，绕过这类基于客户端特征的拦截。
• 核对请求方法与参数：你代码中写的是POST请求，但报错栈显示最终触发错误的是GET请求，先确认目标接口支持的请求方法。如果curl用GET请求能通，先把代码中的POST改成GET测试；如果接口需要POST请求，检查是否遗漏了必填的表单参数，空POST请求也会触发站点防护规则。
• 调整请求头与连接参数：可以先删除请求头中的Accept-Encoding字段，或者将Connection头设置为close测试短连接是否能正常请求，部分站点对非浏览器发起的keep-alive长连接、gzip压缩请求会直接断开连接。
• 关闭底层自动处理逻辑：请求时添加verify=False、allow_redirects=True参数，排查是否是证书校验或者重定向处理逻辑触发的连接重置。

内容的提问来源于stack exchange，提问作者Naveed