如何用requests和BeautifulSoup爬取WSJ标题？页面访问失败求助

核心问题分析 #

你遇到的「无法找到页面」提示，本质是WSJ的反爬机制识别出请求来自脚本而非真实浏览器，或者目标URL已失效。下面是针对性的修复方案：

1. 添加浏览器请求头模拟真实访问 #

WSJ会校验请求的User-Agent等标识，直接用requests.get会被拦截。修改请求部分，加入完整的请求头：

def get_wsj_news():
    global prev_news_wsj
    url = "https://www.wsj.com/world"  # 确认当前世界新闻板块的正确URL，原URL可能已变更
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36",
        "Accept-Language": "en-US,en;q=0.9",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
    }
    news = []
    news_to_post = []
    try:
        response = requests.get(url, headers=headers)
        response.raise_for_status()  # 主动抛出HTTP错误，比如403/404
        soup = BeautifulSoup(response.content, "html.parser")
        # 用通用逻辑匹配标题标签，避免因class更新失效
        news_list = soup.find_all("h3", class_=lambda x: x and "headline" in x.lower())
        for item in news_list[:15]:
            headline = item.text.strip()
            if headline:  # 过滤空标题
                news.append(f"• {headline}")
                news_to_post.append(f"• <b>{headline}</b>")
        if not news or news == prev_news_wsj:
            return {"site": None, "message": None}
        else:
            prev_news_wsj = news.copy()  # 用copy避免引用问题
            print(url)
            return {"site": "The Wall Street Journal", "message": "\n".join(news_to_post)}
    except Exception as e:
        print(f"爬取错误: {str(e)}")
        return {"site": None, "message": None}

2. 确认目标URL的有效性 #

原代码中的https://www.wsj.com/news/world可能已失效，当前WSJ世界新闻板块的正确URL通常是https://www.wsj.com/world。你可以手动打开浏览器访问确认后，再替换到代码中。

3. 应对进阶反爬（如Cloudflare验证） #

如果添加请求头后仍被拦截，大概率遇到了Cloudflare的人机验证。这种情况下requests无法直接处理，需要改用selenium模拟真实浏览器操作：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options

def get_wsj_news_selenium():
    global prev_news_wsj
    url = "https://www.wsj.com/world"
    chrome_options = Options()
    chrome_options.add_argument("--headless=new")  # 无头模式
    chrome_options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36")
    
    driver = webdriver.Chrome(options=chrome_options)
    news = []
    news_to_post = []
    try:
        driver.get(url)
        driver.implicitly_wait(10)  # 等待页面加载
        news_list = driver.find_elements(By.TAG_NAME, "h3")
        for item in news_list[:15]:
            headline = item.text.strip()
            if headline:
                news.append(f"• {headline}")
                news_to_post.append(f"• <b>{headline}</b>")
        if not news or news == prev_news_wsj:
            return {"site": None, "message": None}
        else:
            prev_news_wsj = news.copy()
            print(url)
            return {"site": "The Wall Street Journal", "message": "\n".join(news_to_post)}
    except Exception as e:
        print(f"爬取错误: {str(e)}")
        return {"site": None, "message": None}
    finally:
        driver.quit()

4. 动态调整标签选择器 #

WSJ的页面class名称经常更新，原代码中的WSJTheme--headline--unZqjb45大概率已失效。建议用更通用的选择逻辑：

• 用class_=lambda x: x and "headline" in x.lower()匹配包含headline的class
• 或者直接通过标签层级定位，比如soup.select("article h3")选择文章内的h3标签

内容的提问来源于stack exchange，提问作者miraculous_guru