Python newspaper库无法提取部分URL的文章发布日期问题求助

• 现象：部分可正常访问（状态码200）的URL无法提取发布日期，同一域名下其他链接可正常提取，排除IP封禁，单独测试该URL仍失败
• 测试URL：https://www.aljazeera.com/program/featured-documentaries/2020/12/29/lords-of-water-episode-one

测试代码 #

import os
import sys
from newspaper import Article   

def split(link):
        try:
            story = Article(link)
            story.download()
            story.parse()
            date_time = str(story.publish_date)
            split_date = date_time.split()  
            date = split_date[0]
            if date != "None":
                print(date)
        except:
            print("This URL did not return a published date. Try a different URL.")
            print(link)

if __name__ == "__main__":
        link = "https://www.aljazeera.com/program/featured-documentaries/2020/12/29/lords-of-water-episode-one"
        split(link)

运行输出 #

This URL did not return a published date. Try a different URL.
https://www.aljazeera.com/program/featured-documentaries/2020/12/29/lords-of-water-episode-one

解决方法 #

1. 手动解析页面元数据或HTML元素 #

newspaper的自动解析依赖标准页面元标签，部分页面的日期可能存放在非标准位置，可直接解析页面源码提取：

import os
import sys
from newspaper import Article
from bs4 import BeautifulSoup
import re

def split(link):
        try:
            story = Article(link)
            story.download()
            story.parse()
            
            # 优先尝试newspaper自动提取
            if story.publish_date:
                print(story.publish_date.strftime("%Y-%m-%d"))
                return
            
            # 手动解析Al Jazeera页面的meta标签
            soup = BeautifulSoup(story.html, "html.parser")
            date_meta = soup.find("meta", property="article:published_time")
            if date_meta:
                print(date_meta["content"].split("T")[0])
                return
            
            # 备选：从URL路径提取日期（该URL包含明确日期段）
            date_match = re.search(r"/(\d{4})/(\d{2})/(\d{2})/", link)
            if date_match:
                print(f"{date_match.group(1)}-{date_match.group(2)}-{date_match.group(3)}")
                return
            
            print("无法提取发布日期")
            print(link)
        except Exception as e:
            print(f"处理出错: {str(e)}")
            print(link)

if __name__ == "__main__":
        link = "https://www.aljazeera.com/program/featured-documentaries/2020/12/29/lords-of-water-episode-one"
        split(link)

2. 优化工具配置与版本 #

• 升级至最新版newspaper3k：pip install --upgrade newspaper3k
• 针对目标域名编写自定义提取规则，扩展库的解析逻辑，减少对通用规则的依赖。

3. 异常处理优化 #

原代码用except:捕获所有异常，不利于排查问题，建议改为捕获特定异常并打印错误信息，准确定位故障原因。

内容的提问来源于stack exchange，提问作者Sam Hall