WebScraping项目报错：AttributeError: 'NoneType'无find属性的排查

• 在循环中加入异常捕获机制，当出现AttributeError时打印当前出错的URL，直接定位问题链接：

  for idx, URL in enumerate(URLs):
      try:
          page = requests.get(URL)
          soup = BeautifulSoup(page.content, 'html.parser')
          
          project_title = soup.find('h1').text
          project_description = soup.find('div', class_="column-1").text
          project_details = soup.find(class_="block-details")
          project_number = project_details.find("strong").text
          # 其他爬取逻辑...
          
          data.append((project_title, project_description, project_number, project_start, project_end, project_program, project_recipient))
      except AttributeError as e:
          print(f"第{idx+1}个链接出错: {URL}")
          print(f"错误详情: {e}")
          continue
  

• 也可以提前做空值判断，调用find前检查元素是否存在，避免触发异常：

  for URL in URLs:
      page = requests.get(URL)
      soup = BeautifulSoup(page.content, 'html.parser')
      
      project_details = soup.find(class_="block-details")
      if not project_details:
          print(f"页面无block-details元素: {URL}")
          continue
      
      project_number_elem = project_details.find("strong")
      if not project_number_elem:
          print(f"页面无strong元素: {URL}")
          continue
      project_number = project_number_elem.text
      # 其他字段同理逐个判断
  

1. 完善异常处理：所有soup.find()操作都可能返回None，必须逐个判断，避免单个字段缺失导致爬取中断。
2. 优化请求配置：
   • 添加请求头模拟浏览器，降低被拦截概率：

     headers = {
         'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'
     }
     page = requests.get(URL, headers=headers)
     

   • 使用requests.Session()复用连接，提升请求效率：

     session = requests.Session()
     session.headers.update(headers)
     # 循环内用session.get(URL)代替requests.get(URL)
     

   • 添加随机延时，避免频繁请求给服务器造成压力：

     import random
     time.sleep(random.uniform(1, 3))  # 每次请求后等待1-3秒
     

3. 数据处理更严谨：
   • 用get_text(strip=True)替代.text，自动去除多余空格和换行：

     project_title = soup.find('h1').get_text(strip=True) if soup.find('h1') else '无标题'
     

   • 处理keywords时，避免因找不到对应<p>标签导致变量未定义：

     keywords = []
     for node in project_details.find_all("p"):
         keywords = node.text.split(', ')
     project_recipient = keywords[-1].strip() if keywords else '无接收方'
     

4. 代码结构模块化：把单页面爬取逻辑封装成函数，提升可读性和可维护性：

   def scrape_project(session, url):
       try:
           page = session.get(url)
           page.raise_for_status()  # 捕获404、500等HTTP错误
           soup = BeautifulSoup(page.content, 'html.parser')
           
           project_title = soup.find('h1').get_text(strip=True) if soup.find('h1') else '无标题'
           project_desc_elem = soup.find('div', class_="column-1")
           project_description = project_desc_elem.get_text(strip=True) if project_desc_elem else '无描述'
           
           project_details = soup.find(class_="block-details")
           if not project_details:
               return None
           
           project_number = project_details.find("strong").get_text(strip=True) if project_details.find("strong") else '无项目编号'
           project_start = project_details.find("span", class_="bar-start").get_text(strip=True) if project_details.find("span", class_="bar-start") else '无开始时间'
           project_end = project_details.find("span", class_="bar-end").get_text(strip=True) if project_details.find("span", class_="bar-end") else '无结束时间'
           project_program_elem = project_details.find("ul")
           project_program = project_program_elem.get_text(strip=True, separator='\n') if project_program_elem else '无项目计划'
           
           keywords = []
           for node in project_details.find_all("p"):
               keywords = node.text.split(', ')
           project_recipient = keywords[-1].strip() if keywords else '无接收方'
           
           return (project_title, project_description, project_number, project_start, project_end, project_program, project_recipient)
       except Exception as e:
           print(f"爬取{url}失败: {e}")
           return None
   
   # 主逻辑调用
   session = requests.Session()
   session.headers.update(headers)
   for url in URLs:
       result = scrape_project(session, url)
       if result:
           data.append(result)
       time.sleep(random.uniform(1, 3))
   

5. 捕获HTTP错误：加入page.raise_for_status()，处理请求失败的情况，避免解析无效页面内容。

内容的提问来源于stack exchange，提问作者almondmatchabiscotti