Python中urllib3.LocationParseError与requests.InvalidURL错误解决

尝试向指定URL发送GET请求排查网页访问问题，运行代码时触发urllib3.exceptions.LocationParseError和requests.exceptions.InvalidURL错误，已尝试执行pip3 install six==1.15.0升级，问题仍存在。

错误栈信息 #

Traceback (most recent call last):
  File "C:\Users\dembi\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\models.py", line 434, in prepare_url
    scheme, auth, host, port, path, query, fragment = parse_url(url)
  File "C:\Users\dembi\AppData\Local\Programs\Python\Python39\lib\site-packages\urllib3\util\url.py", line 397, in parse_url
    return six.raise_from(LocationParseError(source_url), None)
  File "<string>", line 3, in raise_from
urllib3.exceptions.LocationParseError: Failed to parse: <Request [GET]>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\dembi\OneDrive\Desktop\CyPros\projects\1 - WebAdsAuto\Main.py", line 5, in <module>
    print(Website.readingPage("https://dash.bidmind.com/dashboard"))
  File "C:\Users\dembi\OneDrive\Desktop\CyPros\projects\1 - WebAdsAuto\Website.py", line 9, in readingPage
    response = requests.get(request)
  File "C:\Users\dembi\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "C:\Users\dembi\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Users\dembi\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\sessions.py", line 573, in request
    prep = self.prepare_request(req)
  File "C:\Users\dembi\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\sessions.py", line 484, in prepare_request
    p.prepare(
  File "C:\Users\dembi\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\models.py", line 368, in prepare
    self.prepare_url(url, params)
  File "C:\Users\dembi\AppData\Local\Programs\Python\Python39\lib\site-packages\requests\models.py", line 436, in prepare_url
    raise InvalidURL(*e.args)
requests.exceptions.InvalidURL: Failed to parse: <Request [GET]>

原始代码 #

import requests
from bs4 import BeautifulSoup

class Website:
    @staticmethod
    def readingPage(url: str) -> list:
        pages = []
        request = requests.Request("GET", url)
        response = requests.get(request)
        if response.status_code == 200:
            soup = BeautifulSoup(response.content, "html.parser")
            for page in soup.select("tbody tr"):
                pages.append(page)
        else:
            print("Error: " + str(response.status_code))
        return pages

print(Website.readingPage("https://dash.bidmind.com/dashboard"))

解决方案 #

错误原因 #

requests.get()方法的第一个参数要求传入URL字符串，但你错误地传入了requests.Request对象。库内部尝试解析该对象的字符串表示<Request [GET]>作为URL，导致解析失败，抛出InvalidURL错误。升级six包无法解决问题，因为这是代码逻辑错误，不是依赖版本问题。

修复方案1：直接使用requests.get(url)（推荐） #

这是requests库发送GET请求的标准方式，无需手动构建Request对象：

import requests
from bs4 import BeautifulSoup

class Website:
    @staticmethod
    def readingPage(url: str) -> list:
        pages = []
        response = requests.get(url)  # 直接传入URL字符串
        if response.status_code == 200:
            soup = BeautifulSoup(response.content, "html.parser")
            for page in soup.select("tbody tr"):
                pages.append(page)
        else:
            print("Error: " + str(response.status_code))
        return pages

print(Website.readingPage("https://dash.bidmind.com/dashboard"))

修复方案2：使用Session发送预构建的Request对象 #

如果需要自定义请求头、参数等，必须通过Session对象发送预准备的Request：

import requests
from bs4 import BeautifulSoup

class Website:
    @staticmethod
    def readingPage(url: str) -> list:
        pages = []
        req = requests.Request("GET", url)
        prepared_req = req.prepare()  # 预准备请求
        # 通过Session发送请求
        with requests.Session() as session:
            response = session.send(prepared_req)
        if response.status_code == 200:
            soup = BeautifulSoup(response.content, "html.parser")
            for page in soup.select("tbody tr"):
                pages.append(page)
        else:
            print("Error: " + str(response.status_code))
        return pages

print(Website.readingPage("https://dash.bidmind.com/dashboard"))

内容的提问来源于stack exchange，提问作者Bar Dembin