如何用NLTK或纯Python从大CSV特定列生成并统计Unigram、Bigram、Trigram频率

Hey there! I see you already have code that generates Unigrams, Bigrams, and Trigrams from text—great start! Now you want to adapt that to pull text from a specific column in a large CSV file and count N-gram frequencies. Let's walk through how to do this efficiently, with practical code examples that play nice with your existing logic.

First, large files mean we can't load everything into memory at once. We'll use streaming/chunked processing to avoid memory overload. Two solid approaches here: using Python's built-in csv module (lightweight, no extra dependencies) or pandas with chunking (great if you're already using pandas for data work).

Assuming your current code has a function that takes text and an N-value, and returns a list of N-grams, we'll plug that into our CSV processing workflow. Let's start with a placeholder for your existing function (swap this out with your actual code from the screenshot!):

import re

def generate_ngrams(text, n):
    # Replace this with your existing N-gram generation logic
    # Example preprocessing & generation:
    text = text.lower().strip()
    text = re.sub(r'[^\w\s]', '', text)  # Remove punctuation (adjust as needed)
    tokens = text.split()
    
    ngrams = []
    for i in range(len(tokens) - n + 1):
        ngram = ' '.join(tokens[i:i+n])
        ngrams.append(ngram)
    return ngrams

This is the most memory-efficient option for huge files. We'll read the CSV row-by-row, extract the target column text, generate N-grams, and accumulate frequencies with collections.Counter.

import csv
from collections import Counter

def process_csv_ngrams(csv_file_path, target_column, n_values=[1,2,3]):
    # Initialize counters for each N-gram type
    ngram_counters = {n: Counter() for n in n_values}
    
    with open(csv_file_path, 'r', encoding='utf-8') as csv_file:
        reader = csv.DictReader(csv_file)
        
        # Validate the target column exists
        if target_column not in reader.fieldnames:
            raise ValueError(f"Column '{target_column}' not found in the CSV.")
        
        # Process each row one at a time
        for row in reader:
            text = row[target_column].strip()
            if not text:
                continue  # Skip empty text entries
            
            # Generate and count N-grams for each specified N
            for n in n_values:
                ngrams = generate_ngrams(text, n)
                ngram_counters[n].update(ngrams)
    
    return ngram_counters

# Example usage
if __name__ == "__main__":
    results = process_csv_ngrams(
        csv_file_path="your_large_file.csv",
        target_column="your_text_column_name"
    )
    
    # Print top 10 Unigrams
    print("Top 10 Unigrams:")
    for gram, count in results[1].most_common(10):
        print(f"{gram}: {count}")
    
    # Print top 10 Bigrams
    print("\nTop 10 Bigrams:")
    for gram, count in results[2].most_common(10):
        print(f"{gram}: {count}")
    
    # Print top 10 Trigrams
    print("\nTop 10 Trigrams:")
    for gram, count in results[3].most_common(10):
        print(f"{gram}: {count}")

If you prefer pandas for data handling, use chunksize to process the CSV in smaller batches. This is more convenient if you need to do additional data cleaning on the column.

import pandas as pd
from collections import Counter

def process_large_csv_pandas(csv_file_path, target_column, chunksize=10000, n_values=[1,2,3]):
    ngram_counters = {n: Counter() for n in n_values}
    
    # Iterate over CSV chunks
    for chunk in pd.read_csv(csv_file_path, chunksize=chunksize):
        # Drop rows with missing values in the target column
        valid_texts = chunk[target_column].dropna().astype(str).str.strip()
        
        for text in valid_texts:
            if not text:
                continue
            for n in n_values:
                ngrams = generate_ngrams(text, n)
                ngram_counters[n].update(ngrams)
    
    return ngram_counters

# Example usage is the same as the csv module approach!

• Text Preprocessing: Adjust the generate_ngrams function to match your needs—add stopword removal (using nltk.corpus.stopwords), handle special characters, or normalize whitespace as needed.
• Memory Optimization: If your CSV is extremely large, reduce the chunksize in pandas or stick with the native csv module.
• Speed Boost: For very large datasets, consider using multiprocessing to parallelize chunk processing. Just make sure to use a thread-safe counter (like multiprocessing.Manager().Counter()).
• Validate Your Input: Add checks for empty rows, non-string values in the target column, or encoding issues to avoid crashes.

内容的提问来源于stack exchange，提问作者user4296720