Python3.8 urllib读取本地服务器GET请求无返回数据问题求助

问题背景 #

在Python 3.8中向本地服务器（192.168.3.1:8080）发送基础GET请求，能成功建立连接并返回200状态码，但resp.read()始终得到空字节；而Python 2.7的近乎相同代码却能正常获取完整HTML响应。Wireshark抓包显示服务器对两个版本的请求均返回了完整HTML内容。

Python 3.8测试代码及调试信息 #

代码 #

import http.client
import urllib.request

def getData():
    data = ""
    strEPAaddress = '192.168.3.1'
    http.client._MAXHEADERS = 1000
    try:
        handler = urllib.request.HTTPHandler(debuglevel=10)
        opener = urllib.request.build_opener(handler)
        resp = opener.open("http://%s:8080"%(strEPAaddress))
        data = resp.read()
    except ConnectionError as e:
        print("Connection Error: {}".format(e))
    except Exception as e:
        print("Error: {}".format(e))

调试输出 #

send: b'GET / HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: 192.168.3.1:8080\r\nUser-Agent: Python-urllib/3.8\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: WebSilicon HTTPd
header: Last-modified: Tue, 18 Nov 2007 13:09:42 GMT
header: Cache-Control: no-store
header: Connection: close
[('Server', 'WebSilicon HTTPd'), ('Last-modified', 'Tue, 18 Nov 2007 13:09:42 GMT'), ('Cache-Control', 'no-store'), ('Connection', 'close')]

b''

Python 2.7代码及调试信息 #

代码 #

import urllib, httplib

def getData():
    data = ""
    strEPAaddress = '192.168.3.1'
    try:
        httplib.HTTPConnection.debuglevel = 1
        resp = urllib.urlopen("http://%s:8080"%(strEPAaddress))
        data = resp.read()

调试输出 #

send: 'GET / HTTP/1.0\r\nHost: 192.168.3.1:8080\r\nUser-Agent: Python-urllib/1.17\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: WebSilicon HTTPd
header: Last-modified: Tue, 18 Nov 2007 13:09:42 GMT
header: Cache-Control: no-store
header: Connection: close

<TITLE>**********</TITLE>
</HEAD>
<BODY bgcolor=white topmargin=0>
<P align=center><br>
<font face=arial size=6 color=Black><B>****** Status Summary</B></font><br><br>
<TABLE WIDTH=100% BORDERCOLOR=Black ALIGN=center BORDER=1 CELLSPACING=0 CELLPADDING=0>
<TABLE WIDTH=90% BORDERCOLOR=Black ALIGN=center BORDER=1 CELLSPACING=0 CELLPADDING=0>
<TR>
    <TD bgColor="75b4b4" width="90%"><font size=4>&nbsp;<B>System Info</B></font></TD>
</TABLE>
<TABLE WIDTH=90% BORDERCOLOR=Black ALIGN=center BORDER=1 CELLSPACING=0 CELLPADDING=0>
<TR>
    <TD bgColor="9EB7B4"  width="25%">&nbsp;Manufacturer</TD>
-----------

排查方向与解决方案 #

核心差异分析 #

对比两个版本的请求头，关键区别在于：

• Python 3.8使用HTTP/1.1，自动添加了Accept-Encoding: identity头，User-Agent为Python-urllib/3.8
• Python 2.7使用HTTP/1.0，无Accept-Encoding头，User-Agent为Python-urllib/1.17

服务器是2007年的WebSilicon HTTPd，对HTTP/1.1的支持可能存在兼容性问题，比如无法正确处理Accept-Encoding头，或者在HTTP/1.1协议下未正确返回响应体。

解决方案1：模拟Python 2.7的请求环境 #

修改Python 3.8代码，使用HTTP/1.0协议，移除Accept-Encoding头，替换为旧版User-Agent：

import urllib.request

def getData():
    data = b""
    strEPAaddress = '192.168.3.1'
    try:
        # 构造请求，模拟Python 2的请求参数
        req = urllib.request.Request(f"http://{strEPAaddress}:8080", method='GET')
        req.add_header('User-Agent', 'Python-urllib/1.17')
        req.add_header('Connection', 'close')
        # 指定使用HTTP/1.0
        req.version = 10
        # 移除默认的Accept-Encoding头
        req.remove_header('Accept-Encoding')
        
        with urllib.request.urlopen(req) as resp:
            data = resp.read()
        print(data.decode('utf-8'))
    except ConnectionError as e:
        print(f"Connection Error: {e}")
    except Exception as e:
        print(f"Error: {e}")

getData()

解决方案2：直接使用http.client手动处理连接 #

如果urllib的封装仍有问题，可直接用http.client底层接口：

import http.client

def getData():
    data = b""
    strEPAaddress = '192.168.3.1'
    http.client._MAXHEADERS = 1000
    try:
        conn = http.client.HTTPConnection(strEPAaddress, 8080)
        conn.set_debuglevel(10)
        # 发送HTTP/1.0请求，完全匹配Python 2的请求头
        conn.request(
            method="GET",
            url="/",
            headers={
                'Host': f"{strEPAaddress}:8080",
                'User-Agent': 'Python-urllib/1.17',
                'Connection': 'close'
            },
            version=10  # HTTP/1.0
        )
        resp = conn.getresponse()
        data = resp.read()
        print(data.decode('utf-8'))
        conn.close()
    except ConnectionError as e:
        print(f"Connection Error: {e}")
    except Exception as e:
        print(f"Error: {e}")

getData()

其他排查点 #

• 检查服务器是否在HTTP/1.1下使用了分块传输（Transfer-Encoding: chunked），但Python 3的解析逻辑出现问题；可尝试手动读取响应的每一块数据
• 确认服务器返回的响应体编码，尝试指定编码后再读取

内容的提问来源于stack exchange，提问作者Dave