newspaper3k库article.authors方法无法提取新闻作者的问题求助

Hey there! This is a super common issue when using newspaper3k for author extraction, and it’s almost certainly not a mistake in your basic workflow. Let me walk through why this happens and how you can improve your results:

Why Authors Go Missing #

newspaper3k relies on predefined rules and HTML structure patterns to pull out metadata like authors. The problem is:

• Every news site structures its HTML differently—some put authors in <meta> tags, others in inline spans, divs, or even hidden elements.
• Some sites load author content dynamically with JavaScript, which newspaper3k doesn’t process by default.
• A few sites intentionally obfuscate author info to prevent scraping.

Fixes & Workarounds to Try #

1. Ensure Proper Parsing Workflow (Double-Check Your Batch Code) #

First, make sure you’re completing the full download/parse cycle for each article, and adding delays to avoid incomplete downloads or anti-scraping blocks. Here’s a solid batch template:

import time
from newspaper import Article

url_list = ["your-url-1", "your-url-2", ...]

for idx, url in enumerate(url_list):
    article = Article(url)
    try:
        # Download and parse the article fully
        article.download()
        article.parse()
        
        # Extract authors
        authors = article.authors
        print(f"Article {idx+1} authors: {authors if authors else 'No authors found'}")
        
        # Add a small delay to avoid overwhelming servers
        time.sleep(1.5)
    except Exception as e:
        print(f"Failed to process article {idx+1} ({url}): {str(e)}")

2. Add Custom Extraction Rules for Problem Sites #

If you notice specific sites consistently missing authors, you can manually target their HTML structure. For example, if a site uses <div class="article-author"> for authors:

# After parsing the article
custom_authors = []
# Use the extractor to target specific elements
author_elements = article.extractor.get_elements_by_tag('div', attrs={'class': 'article-author'})
for elem in author_elements:
    if elem.text and elem.text.strip():
        custom_authors.append(elem.text.strip())

# Fall back to newspaper3k's default if custom extraction fails
final_authors = custom_authors if custom_authors else article.authors

3. Handle JavaScript-Rendered Content #

For sites that load authors via JS, you’ll need to render the page first. Use a tool like Selenium to get the fully loaded HTML, then pass it to newspaper3k:

from selenium import webdriver
from newspaper import Article

# Initialize Selenium (make sure you have ChromeDriver installed)
driver = webdriver.Chrome()

for url in url_list:
    driver.get(url)
    # Wait a second for JS to load (adjust if needed)
    time.sleep(2)
    # Get the fully rendered HTML
    page_html = driver.page_source
    
    # Pass the HTML to newspaper3k
    article = Article(url)
    article.set_html(page_html)
    article.parse()
    
    print(f"Authors for {url}: {article.authors}")

driver.quit()

Final Note #

newspaper3k is a great tool but isn’t perfect for every edge case. Mixing its default extraction with custom rules and JS rendering will drastically improve your author capture rate.

内容的提问来源于stack exchange，提问作者tursunWali