使用BeautifulSoup和Requests过滤mailto链接失败，求解决方案

问题场景 #

尝试用BeautifulSoup+Requests编写爬虫，意图过滤mailto://和mailto:格式的链接，但爬取部分网页时仍触发InvalidSchema错误，错误提示找不到mailto://链接的适配连接器。

原过滤代码 #

for link in links:
    if not link.startswith('mailto://'):
        if not link.startswith('mailto:'):
            if total_urls_visited > max_urls:
                break
            crawl(link, max_urls=max_urls)
            if not link.startswith('GET'):
                continue
            elif link.endswith(ignore_list): #use ignore directly here
                continue
            else:
                print(link)

错误追踪信息 #

Traceback (most recent call last):
  File "/Users/user/Desktop/site_crawler/get_all_links.py", line 175, in <module>
    reqs = requests.get(internal_link, headers=headers)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/sessions.py", line 695, in send
    adapter = self.get_adapter(url=request.url)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/sessions.py", line 792, in get_adapter
    raise InvalidSchema(f"No connection adapters were found for {url!r}")
requests.exceptions.InvalidSchema: No connection adapters were found for 'mailto://info@mail.app

有效解决方法 #

1. 调整过滤逻辑顺序，提前拦截无效链接 #

原代码的问题是：在判断完非mailto链接后直接调用了crawl，后续的过滤逻辑（如startswith('GET')）在爬取之后执行，导致无效链接已经被传入requests.get()触发错误。

修改后代码：

for link in links:
    # 优先过滤所有mailto格式链接
    if link.startswith(('mailto:', 'mailto://')):
        continue
    # 检查爬取数量限制
    if total_urls_visited > max_urls:
        break
    # 过滤其他不符合要求的链接
    if not link.startswith('GET'):
        continue
    if link.endswith(ignore_list):
        continue
    # 执行爬取和输出
    crawl(link, max_urls=max_urls)
    print(link)

2. 在crawl函数内部增加协议校验 #

为了彻底避免非HTTP/HTTPS协议链接触发错误，在crawl函数调用requests.get()前增加协议检查：

def crawl(link, max_urls):
    # 只处理HTTP/HTTPS协议的链接
    if not link.startswith(('http://', 'https://')):
        return
    # 原有爬取逻辑
    reqs = requests.get(link, headers=headers)
    content_type = reqs.headers.get('content-type')
    # ... 后续代码

3. 用urlparse标准化解析链接（可选） #

针对格式不规范的mailto链接，可通过urlparse准确识别协议类型：

from urllib.parse import urlparse

for link in links:
    parsed_link = urlparse(link)
    # 协议为mailto则直接跳过
    if parsed_link.scheme == 'mailto':
        continue
    # 后续过滤及爬取逻辑
    # ...

内容的提问来源于stack exchange，提问作者mem