I totally get your frustration—those repetitive journal names, author lists, and standardized terms in references can really skew topic model results, especially when dealing with heterogeneous PDFs from different journals. Simple keyword matching often falls flat here, so let’s walk through some industry best practices you can adapt:

1. Refine Your Keyword-Based Truncation (Your Idea, Supercharged) #

Your instinct to target the last occurrence of reference markers is spot-on—this avoids cutting off valid content early. But you’ll need to expand your keyword list to cover all the variants journals use. Here’s how to implement this robustly in Python:

import re

def strip_references(text):
    # Cover common reference section headers, including formatting variations
    reference_headers = [
        r'\bReferences\b',
        r'\bBibliography\b',
        r'\bWorks Cited\b',
        r'\bLiterature Cited\b',
        r'\b\d+\.\s*References\b',  # For numbered sections like "1. References"
        r'\bReferences\s*\d*\b'     # For headers with optional numbers
    ]
    # Compile a case-insensitive regex to match any of these
    header_pattern = re.compile('|'.join(reference_headers), re.IGNORECASE)
    
    # Find all matches in the text
    matches = list(header_pattern.finditer(text))
    if matches:
        # Grab the end position of the LAST match to avoid early truncation
        last_ref_start = matches[-1].end()
        # Truncate and clean up trailing whitespace/empty lines
        return text[:last_ref_start].strip()
    # If no references found, return original text
    return text

Pro tip: After truncating, run a quick cleanup to remove any leftover boilerplate (like copyright notices or blank pages) at the end of the text.

2. Leverage PDF Structural Metadata (Most Reliable When Available) #

Many academic PDFs come with built-in structure—bookmarks, outlines, or tagged content—that lets you directly extract just the main body, skipping references entirely. Tools like PyMuPDF (fitz) or pdfplumber can access this metadata:

import fitz  # PyMuPDF

def extract_main_content(doc):
    main_text = ""
    reference_section_found = False
    
    # Check for table of contents/bookmarks first
    for toc_item in doc.get_toc():
        level, section_title, page_num = toc_item
        # Look for reference-related section names
        if re.search(r'reference|bibliography', section_title.lower()):
            reference_section_found = True
            break
        # Extract text from pages before the reference section
        page = doc.load_page(page_num - 1)  # PyMuPDF uses 0-indexed pages
        main_text += page.get_text()
    
    # Fall back to keyword truncation if no bookmarks exist
    if not reference_section_found:
        main_text = strip_references(main_text)
    
    return main_text

Note: Not all PDFs have structured bookmarks, so always include a fallback to your keyword method.

3. Use NLP Models for Unstructured PDFs #

For PDFs with no discernible structure, pre-trained NLP models can help classify paragraphs as "main text" or "reference." Zero-shot classification works great here if you don’t have labeled data:

from transformers import pipeline

# Load a pre-trained zero-shot classifier
classifier = pipeline("zero-shot-classification", model="facebook/bart-large-mnli")

def filter_reference_paragraphs(text):
    paragraphs = text.split('\n\n')
    filtered_text = []
    
    for para in paragraphs:
        # Classify the paragraph
        classification = classifier(para, candidate_labels=["reference", "main text"])
        # Keep only paragraphs labeled as main text
        if classification['labels'][0] == "main text":
            filtered_text.append(para)
    
    return '\n\n'.join(filtered_text)

This method is more flexible but slower—ideal for small datasets or as a final validation step after rule-based filtering.

4. Combine Rules with Statistical Features #

References have distinct statistical patterns: frequent year markers (like (2023)), DOIs, author name formats (e.g., Smith, J.), and journal abbreviations. You can build rules to flag these:

def is_reference_paragraph(para):
    # Count occurrences of common reference patterns
    year_matches = len(re.findall(r'\(\d{4}\)', para))
    doi_matches = len(re.findall(r'doi:\d+\.\d+/\S+', para, re.IGNORECASE))
    author_matches = len(re.findall(r'[A-Z][a-z]+,\s+[A-Z]\.', para))
    
    # If any pattern is over a threshold, flag as reference
    return year_matches > 3 or doi_matches > 0 or author_matches > 2

def filter_by_statistics(text):
    paragraphs = text.split('\n\n')
    return '\n\n'.join([p for p in paragraphs if not is_reference_paragraph(p)])

Pair this with your keyword method for an extra layer of accuracy.

Final Recommendation #

Start with structured metadata extraction + refined keyword truncation—this is fast and works for most well-formatted academic PDFs. For messy, unstructured PDFs, add the statistical rule filter as a middle step, and use the NLP classifier only for edge cases or small batches.

内容的提问来源于stack exchange，提问作者Christopher