Newspaper3k指定User Agent后部分新闻站点无法提取内容问题咨询

问题原因 #

• 目标站点eluniversal.com.mx的反爬策略比milenio.com更严格，仅配置User Agent不足以通过反爬校验，站点返回的是拦截页面而非正常新闻内容，因此Newspaper3k无法提取有效字段
• Newspaper3k的内置提取规则是通用适配逻辑，该站点的页面结构、字段存储的标签属性不符合通用规则，导致内置解析逻辑匹配不到对应内容

适配处理方案 #

1. 完善请求配置 #

除User Agent外，补充其他常规请求头字段、开启cookie支持、设置合理超时，降低被反爬拦截的概率：

import time, os, random, nltk, newspaper 
from newspaper import Article, Config

user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124  Safari/537.36'
config = Config()
config.browser_user_agent = user_agent
# 新增配置
config.request_timeout = 15
config.fetch_images = False
config.follow_meta_refresh = True
# 补充完整请求头
config.custom_headers = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
    'Accept-Language': 'es-ES,es;q=0.8,en-US;q=0.5,en;q=0.3',
    'Accept-Encoding': 'gzip, deflate, br',
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1',
    'Sec-Fetch-Dest': 'document',
    'Sec-Fetch-Mode': 'navigate',
    'Sec-Fetch-Site': 'none',
    'Sec-Fetch-User': '?1'
}

url = 'https://www.eluniversal.com.mx/estados/matan-3-policias-durante-ataque-en-nochistlan-zacatecas'
article = Article(url, config=config)
try:
    article.download()
    if article.download_state != 2: # 2代表下载成功
        print("页面下载失败，返回的是拦截内容")
    article.parse()
    article.nlp()
    print(article.authors)
    print(article.publish_date)
    print(article.text)
except Exception as e:
    print(f"处理异常：{e}")

2. 绕过人机验证 #

如果上述配置后还是返回拦截页面，说明站点启用了Cloudflare等动态人机校验，需要用cloudscraper库先自行获取页面源码，再交给Newspaper3k解析：
首先安装依赖：pip install cloudscraper beautifulsoup4
调整后的代码逻辑：

import cloudscraper
from newspaper import Article, Config
from bs4 import BeautifulSoup

# 用cloudscraper发起请求绕过人机验证
scraper = cloudscraper.create_scraper(browser={'browser': 'chrome', 'platform': 'osx', 'mobile': False})
url = 'https://www.eluniversal.com.mx/estados/matan-3-policias-durante-ataque-en-nochistlan-zacatecas'
resp = scraper.get(url)

# 把获取到的源码喂给Newspaper3k
config = Config()
config.browser_user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124  Safari/537.36'
article = Article(url, config=config)
article.set_html(resp.text)
article.parse()

# 若内置提取还是拿不到字段，自行用bs4解析对应标签补全
soup = BeautifulSoup(resp.text, 'lxml')
# 示例：从meta标签提取发布日期、作者
publish_date = soup.find('meta', property='article:published_time')['content'] if soup.find('meta', property='article:published_time') else None
author = soup.find('meta', property='article:author')['content'] if soup.find('meta', property='article:author') else None
# 示例：提取正文内容
content = soup.select_one('.contenido-nota').get_text(strip=True) if soup.select_one('.contenido-nota') else article.text

3. 自定义规则适配 #

如果经常需要爬取该站点的内容，可以直接基于站点的页面结构写固定的xpath/css提取规则，比Newspaper3k的通用规则准确率更高，效率也更快。

内容的提问来源于stack exchange，提问作者Jared Greathouse