Python脚本清理含指定子串URL失效，请求技术排查

问题描述 #

我编写了一个Python脚本爬取新闻通讯中的URL，需要移除fortune.com、forbes.com、twitter.com及mailto:开头的链接，但添加清理逻辑后部分目标URL未被移除。

原代码：

from os import remove
from turtle import clear
from bs4 import BeautifulSoup
import requests
import re
import pandas as pd

termSheet = "https://fortune.com/newsletter/termsheet"
html = requests.get(termSheet)
htmlParser = BeautifulSoup(html.text, "html.parser")
termSheetLinks = []

for companyURL in htmlParser.select("table#templateBody p > a"):
    termSheetLinks.append(companyURL.get('href'))

for link in termSheetLinks:
    if "fortune.com" in link in termSheetLinks:
        termSheetLinks.remove(link)
    if "forbes.com" in link in termSheetLinks:
        termSheetLinks.remove(link)
    if "twitter.com" in link in termSheetLinks:
        termSheetLinks.remove(link)

print(termSheetLinks)

运行输出：

['https://fortune.com/company/blackstone-group?utm_source=email&utm_medium=newsletter&utm_campaign=term-sheet&utm_content=2022080907am', 'https://fortune.com/company/tpg?utm_source=email&utm_medium=newsletter&utm_campaign=term-sheet&utm_content=2022080907am', 'https://casproviders.org/asd-guidelines/', 'https://fortune.com/company/carlyle-group?utm_source=email&utm_medium=newsletter&utm_campaign=term-sheet&utm_content=2022080907am', 'https://ir.carlyle.com/static-files/433abb19-8207-4632-b173-9606698642e5', 'mailto:termsheet@fortune.com', 'https://www.afresh.com/', 'https://www.geopagos.com/', 'https://montana-renewables.com/', 'https://descarteslabs.com/', 'https://www.dealer-pay.com/', 'https://www.sequeldm.com/', 'https://pueblo-mechanical.com/', 'https://dealcloud.com/future-proof-your-firm/', 'https://apartmentdata.com/', 'https://www.irobot.com/', 'https://www.martin-bencher.com/', 'https://cell-matters.com/', 'https://www.lever.co/', 'https://www.sigulerguff.com/']

问题原因 #

1. 遍历列表时直接修改原列表：在for link in termSheetLinks循环中调用remove(link)会导致列表长度动态变化，后续元素的索引被跳过，部分符合条件的链接未被检查到。
2. 条件判断逻辑冗余："fortune.com" in link in termSheetLinks写法多余，link本身就在遍历的列表里，只需检查"fortune.com" in link即可。
3. 未处理mailto:链接：原代码没有针对邮件链接做排除规则。

修复方案 #

使用列表推导式生成过滤后的新列表，避免遍历原列表时修改的问题，同时统一处理所有排除规则：

from bs4 import BeautifulSoup
import requests

termSheet = "https://fortune.com/newsletter/termsheet"
html = requests.get(termSheet)
htmlParser = BeautifulSoup(html.text, "html.parser")
termSheetLinks = []

for companyURL in htmlParser.select("table#templateBody p > a"):
    href = companyURL.get('href')
    if href:  # 过滤空链接
        termSheetLinks.append(href)

# 统一过滤不符合要求的链接
filtered_links = [
    link for link in termSheetLinks
    if not (
        "fortune.com" in link
        or "forbes.com" in link
        or "twitter.com" in link
        or link.startswith("mailto:")
    )
]

print(filtered_links)

改动说明 #

• 移除未使用的导入（remove、clear、re、pd），精简代码。
• 添加if href:判断，避免空值进入链接列表。
• 用列表推导式一次性完成过滤，逻辑清晰且不会出现遍历跳元素的问题。
• 新增link.startswith("mailto:")规则，处理邮件链接。

内容的提问来源于stack exchange，提问作者user18871432