如何解决外部for循环中BeautifulSoup返回NoneType的报错？

问题概况 #

我在开发网站文章爬虫时遇到了异常：单独运行主逻辑或HTML爬取片段都正常，但整合到for循环后，results = soup.find(id="main-content")起初能返回bs4.element.Tag对象，后续却变成NoneType，触发错误：

AttributeError: 'NoneType' object has no attribute 'find_all'

完整代码 #

import newspaper
from newspaper import Article
import pandas as pd
import datetime
from datetime import datetime, timezone
import requests
from bs4 import BeautifulSoup
import re

urls = open("urls_test.txt").readlines()

final_df = pd.DataFrame()

for url in urls:
    article = newspaper.Article(url="%s" % (url), language='en')
    article.download()
    article.parse()
    article.nlp()

    # scrape html part
    page = requests.get(url)
    soup = BeautifulSoup(page.content, "html.parser")
    results = soup.find(id="main-content")
    texts = results.find_all("div", class_="component article-body-text")
    paragraphs = []
    for snippet in texts:
        paragraphs.append(str(snippet))
    CLEANR = re.compile('<.*?>')
    def remove_html(input):
        cleantext = re.sub(CLEANR, '', input)
        return cleantext
    paragraphs_string = ' '.join(paragraphs)
    paragraphs_clean = remove_html(paragraphs_string)
    #

    temp_df = pd.DataFrame(columns=['Title', 'Authors', 'Text', 'Summary', 'published_date', 'URL'])

    temp_df['Authors'] = article.authors
    temp_df['Title'] = article.title
    temp_df['Text'] = paragraphs_clean
    temp_df['Summary'] = article.meta_description
    publish_date = article.publish_date
    publish_date = publish_date.replace(tzinfo=None)
    temp_df['published_date'] = publish_date
    temp_df['URL'] = article.url

    final_df = pd.concat([final_df, temp_df], ignore_index=True)

final_df.to_excel('Telegraph_test.xlsx')

问题原因及修复方案 #

• URL格式异常：readlines()读取的URL可能带有换行符\n或空白字符，导致请求的页面无效，无法找到目标元素。
  修复代码：

  for url in urls:
      url = url.strip()  # 去除换行、空格等空白字符
      if not url:
          continue  # 跳过空行
      # 后续逻辑...
  

• 重复请求触发反爬：代码先通过newspaper.Article请求页面，再用requests.get重复请求同一URL，短时间内多次请求可能被网站反爬机制拦截，返回非目标页面（如404、验证页），导致找不到main-content。
  修复代码：复用newspaper已下载的页面内容，避免重复请求：

  # 替换原scrape html部分的请求代码
  soup = BeautifulSoup(article.html, "html.parser")  # 直接使用article已下载的HTML内容
  results = soup.find(id="main-content")
  

• 页面结构不一致：部分URL对应的文章页面没有id="main-content"的元素，导致find返回None，直接调用find_all触发异常。
  修复代码：增加非空判断，避免空对象调用方法：

  results = soup.find(id="main-content")
  texts = results.find_all("div", class_="component article-body-text") if results else []
  

• 性能优化：函数移出循环：remove_html函数在循环内重复定义会降低性能，建议移到循环外部：

  # 移到循环外，只定义一次
  CLEANR = re.compile('<.*?>')
  def remove_html(input_str):
      cleantext = re.sub(CLEANR, '', input_str)
      return cleantext
  
  for url in urls:
      # 循环内逻辑...
  

内容的提问来源于stack exchange，提问作者Linda Brck