如何用Python提取文本有效段落，借助NLTK实现完整句子主题建模并移除单个词

Hey there! Let's break down how to solve your problem—from cleaning your paper text (excluding tables of contents/headings) to getting proper full sentences for topic modeling, and ditching those annoying single-word fragments.

1. First: Extract Valid Paragraphs (Exclude TOC & Headings) #

First, we need to strip out unwanted sections like the table of contents and isolated headings before even getting to sentence splitting. Here's a practical approach:

• Rule-based filtering: Most papers have a clear "Table of Contents" or "Contents" section—we can skip lines containing these keywords.
• Length-based filtering: Headings are often short, all-caps, or stand alone; we can filter out paragraphs that are too short (e.g., <3 words) or match heading patterns.

Example code:

import nltk
from nltk.corpus import stopwords

# Load your text file
with open("your_paper.txt", "r", encoding="utf-8") as f:
    raw_text = f.read()

# Split text into lines/paragraphs (adjust split based on your text's formatting)
paragraphs = raw_text.split("\n\n")  # assuming paragraphs are separated by double newlines

# Filter out TOC and headings
filtered_paragraphs = []
toc_keywords = ["table of contents", "contents", "abstract", "references"]  # add more as needed

for para in paragraphs:
    # Skip empty paragraphs
    if not para.strip():
        continue
    # Skip TOC sections
    lower_para = para.lower()
    if any(keyword in lower_para for keyword in toc_keywords):
        continue
    # Skip short headings (adjust word count threshold as needed)
    word_count = len(nltk.word_tokenize(para))
    if word_count < 3:
        continue
    filtered_paragraphs.append(para)

# Combine filtered paragraphs into a single clean text block
clean_text = "\n".join(filtered_paragraphs)

2. Fix Sentence Tokenization (Remove Single-Word Fragments) #

Your current Punkt tokenizer works well for most sentences, but it might pick up single words (like page numbers, isolated terms) as "sentences". We'll add a post-filter step to keep only full, valid sentences:

• Check if the "sentence" has at least 2 words.
• Verify it ends with a proper sentence-ending punctuation (., ?, !).

Example code:

# Load the Punkt tokenizer
tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')

# Split clean text into sentences
raw_sentences = tokenizer.tokenize(clean_text)

# Filter out invalid sentences (single words, no proper ending)
valid_sentences = []
sentence_endings = {".", "?", "!"}

for sent in raw_sentences:
    stripped_sent = sent.strip()
    # Skip empty strings
    if not stripped_sent:
        continue
    # Check if it ends with a valid punctuation
    if stripped_sent[-1] not in sentence_endings:
        continue
    # Check word count (keep sentences with >=2 words)
    words = nltk.word_tokenize(stripped_sent)
    if len(words) >= 2:
        valid_sentences.append(stripped_sent)

# Now you have only full, valid sentences!
print("\n".join(valid_sentences))

3. Preprocess Text for Topic Modeling #

Before feeding into a topic model, we need to clean the sentences further (remove stopwords, punctuation, lowercase):

# Download stopwords if you haven't already
nltk.download('stopwords')
stop_words = set(stopwords.words('english'))

# Preprocess each valid sentence
processed_sentences = []
for sent in valid_sentences:
    # Lowercase
    sent_lower = sent.lower()
    # Tokenize words
    words = nltk.word_tokenize(sent_lower)
    # Remove stopwords and non-alphabetic tokens
    filtered_words = [word for word in words if word.isalpha() and word not in stop_words]
    # Only keep sentences that still have content after filtering
    if filtered_words:
        processed_sentences.append(filtered_words)

# Now processed_sentences is ready for topic modeling (e.g., LDA)

4. Sentence-Based Topic Modeling Example #

Using Gensim (a popular library for topic modeling) with your processed sentences:

from gensim import corpora, models

# Create a dictionary and corpus
dictionary = corpora.Dictionary(processed_sentences)
corpus = [dictionary.doc2bow(sent) for sent in processed_sentences]

# Train LDA model (adjust num_topics as needed)
lda_model = models.LdaModel(corpus, num_topics=5, id2word=dictionary, passes=15)

# Print topics
for idx, topic in lda_model.print_topics(-1):
    print(f"Topic {idx+1}: {topic}")

Key Notes: #

• Adjust thresholds (like word count for headings, sentence length) based on your specific paper's formatting—some papers might have longer headings, so tweak accordingly.
• If your paper was converted from a PDF, you might need extra cleaning (remove page numbers, footers) using regex (e.g., re.sub(r'\d+', '', text) to remove numbers).

内容的提问来源于stack exchange，提问作者Jammy