BeautifulSoup .find()仅对部分Yahoo Finance链接生效的问题求助

核心问题 #

1. 缺失依赖导入：代码中使用requests.get()但未导入requests库，运行时会触发NameError。
2. 硬编码元素class导致兼容性问题：Yahoo Finance不同页面的日期容器class存在差异，第一个链接的日期用caas-attr-time-style类包裹，第二个链接则使用caas-meta-time类。原代码直接硬编码查找caas-attr-time-style，导致第二个链接找不到元素，触发AttributeError: 'NoneType' object has no attribute 'find'。

修复步骤 #

1. 补全依赖导入 #

在代码开头添加requests库的导入：

import requests

2. 兼容不同页面的日期查找逻辑 #

修改日期元素的查找逻辑，同时支持两种class，并增加空值判断，避免报错：

# 尝试两种常见的日期容器class，找不到则直接查找time标签
date_container = soup.find(class_='caas-attr-time-style') or soup.find(class_='caas-meta-time')
date = date_container.find('time') if date_container else soup.find('time')

if date:
    dateToAppend = date.get_text().split('at')[0].strip()
    print(dateToAppend)
    file.write(f"Date: {dateToAppend}")
    file.write("\n\n\n\n")
else:
    print(f"无法获取链接 {link} 的日期")
    file.write(f"Date: 无法获取\n\n\n\n")

3. 优化文件操作（可选） #

原代码每次写入都重复打开关闭文件，效率较低。可以将文件操作合并为一次打开，循环内持续写入：

with open(filename, 'a') as file:
    file.write(get_currency_info(curr).to_string())
    file.write("\n\n\n\n")
    
    for link in link_list :
        # 后续的页面抓取、日期写入、文本写入逻辑都放在这个文件上下文内

完整修复代码 #

from bs4 import BeautifulSoup
import datetime
import yfinance as yf
import requests  # 补全缺失的导入

def unique_filename () :
    current_date = datetime.datetime.now()
    day_month_string = current_date.strftime("%S-%M-%H-%d")
    filename = f"{day_month_string}-ScrapedData.txt"
    return filename

def get_currency_info (name) :
    currency = yf.Ticker(name)
    history = currency.history(period="1mo")
    return history

def scrape_all (link_list, curr) :
    filename = unique_filename()
    
    with open(filename, 'a') as file:
        file.write(get_currency_info(curr).to_string())
        file.write("\n\n\n\n")
        
        for link in link_list :
            print("=======================NEW LINK========================")
            source = requests.get(link).text
            soup = BeautifulSoup(source, 'lxml')
            
            # 兼容不同页面的日期元素
            date_container = soup.find(class_='caas-attr-time-style') or soup.find(class_='caas-meta-time')
            date = date_container.find('time') if date_container else soup.find('time')
            
            if date:
                dateToAppend = date.get_text().split('at')[0].strip()
                print(dateToAppend)
                file.write(f"Date: {dateToAppend}")
                file.write("\n\n\n\n")
            else:
                print(f"无法获取链接 {link} 的日期")
                file.write(f"Date: 无法获取\n\n\n\n")
            
            texts = soup.find_all('p')
            for text in texts:
                text_to_append = text.get_text()
                file.write(text_to_append)
                file.write("\n")

if __name__ == "__main__" :
    listOfLinks = [
        'https://uk.finance.yahoo.com/news/bitcoin-byd-bp-marks-spencer-trending-tickers-112621324.html',
        'https://uk.yahoo.com/finance/news/bitcoin-rally-fuelled-by-shrinking-supply-102808887.html'
    ]
    scrape_all(listOfLinks, "BTC")
    print("=======================LIST END========================")

内容的提问来源于stack exchange，提问作者Jakub Zak