捕获urllib的HTTPError时，如何正确读取并打印fp内容？

问题详情 #

使用urllib.request.urlopen()时触发了HTTP 400错误，错误栈如下：

Traceback (most recent call last):
  File "test_urlopen.py", line 47, in <module>
    response = request.urlopen(req, data, context=ctx)
  File "/lib64/python/lib/urllib/request.py", line 227, in urlopen
    return opener.open(url, data, timeout)
  File "/lib64/python/lib/urllib/request.py", line 541, in open
    response = meth(req, response)
  File "/lib64/python/lib/urllib/request.py", line 653, in http_response
    response = self.parent.error(
  File "/lib64/python/lib/urllib/request.py", line 580, in error
    return self._call_chain(*args)
  File "/lib64/python/lib/urllib/request.py", line 508, in _call_chain
    result = func(*args)
  File "/lib64/python/lib/urllib/request.py", line 663, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400:

编写异常捕获代码后，读取e.fp得到空内容：

try:
    response = request.urlopen(req, data, context=ctx)
except urllib.error.HTTPError as e:
    print ("the exception is %s " % str(e.fp))
    print ("the exception is %s " % str(e.fp.readlines()))
    exit(0)

输出结果：

the exception is <http.client.HTTPResponse object at 0xd1b9615640>
the exception is []

但修改HTTPDefaultErrorHandler直接读取fp时，却能获取到内容：

class HTTPDefaultErrorHandler(BaseHandler):
    def http_error_default(self, req, fp, code, msg, hdrs):
        print (str(fp))
        print (fp.readlines())
        raise HTTPError(req.full_url, code, msg, hdrs, fp)

输出结果：

fp <http.client.HTTPResponse object at 0xd1b9615640>
[b'{"code":"6006","reason":"Failed to update".....}']

原因及解决方法 #

原因 #

http.client.HTTPResponse是流式对象，读取操作会移动内部的文件指针。在HTTPDefaultErrorHandler中调用fp.readlines()已经把指针移动到了流的末尾，当HTTPError将这个fp对象传递到异常捕获代码时，再次读取就只能得到空内容。

解决方法 #

有两种可行的处理方式：

1. 在HTTPDefaultErrorHandler抛出HTTPError前，将fp的指针重置到开头：

class HTTPDefaultErrorHandler(BaseHandler):
    def http_error_default(self, req, fp, code, msg, hdrs):
        print (str(fp))
        content = fp.readlines()
        print(content)
        fp.seek(0)  # 重置指针到开头
        raise HTTPError(req.full_url, code, msg, hdrs, fp)

2. 在异常捕获代码中，先重置fp指针再读取：

try:
    response = request.urlopen(req, data, context=ctx)
except urllib.error.HTTPError as e:
    e.fp.seek(0)  # 重置文件指针到开头
    print("the exception is %s " % str(e.fp))
    print("the exception is %s " % str(e.fp.readlines()))
    exit(0)

内容的提问来源于stack exchange，提问作者user2719735