如何使用pos_tags字典替换含'body'列的Pandas DataFrame中的值？

Got it, let's walk through how to replace tokens in your DataFrame's body column using a POS tags dictionary. The core idea is to tag each word with its part-of-speech, then swap it out using your predefined mappings. Here's a step-by-step solution with code examples:

First, Let's Define the Setup #

Assume your pos_tags dictionary maps POS categories (like "NOUN", "VERB") to replacement strings, e.g.:

pos_tags = {
    'NOUN': '[GENERIC_NOUN]',
    'VERB': '[GENERIC_VERB]',
    'ADJ': '[GENERIC_ADJECTIVE]',
    # Add more mappings as needed
}

Option 1: Using NLTK (Simple, Beginner-Friendly) #

NLTK is a classic choice for basic NLP tasks. Let's use it for tokenization and POS tagging.

Step 1: Install & Import Dependencies #

import pandas as pd
import nltk
from nltk.tokenize import word_tokenize
from nltk.tag import pos_tag

# Download required NLTK resources (run once)
nltk.download('punkt')
nltk.download('averaged_perceptron_tagger')

Step 2: Create a Replacement Function #

NLTK returns granular POS tags (e.g., NN for singular noun, NNS for plural), so we'll add a normalization step to map these to broader categories matching your pos_tags keys:

# Map NLTK's granular tags to broader categories
tag_normalizer = {
    # Nouns
    'NN': 'NOUN', 'NNS': 'NOUN', 'NNP': 'NOUN', 'NNPS': 'NOUN',
    # Verbs
    'VB': 'VERB', 'VBD': 'VERB', 'VBG': 'VERB', 'VBN': 'VERB', 'VBP': 'VERB', 'VBZ': 'VERB',
    # Adjectives
    'JJ': 'ADJ', 'JJR': 'ADJ', 'JJS': 'ADJ'
}

def replace_text_with_pos(text, pos_map, tag_norm):
    # Handle empty/NaN values to avoid crashes
    if pd.isna(text) or text.strip() == '':
        return text
    
    # Split text into words
    tokens = word_tokenize(text)
    # Get POS tags for each token
    tagged_tokens = pos_tag(tokens)
    
    # Replace each token: use normalized tag to look up replacement, keep original if no match
    replaced_tokens = []
    for token, tag in tagged_tokens:
        normalized_tag = tag_norm.get(tag, tag)  # Fallback to original tag if no normalization
        replaced_tokens.append(pos_map.get(normalized_tag, token))
    
    # Put the text back together
    return ' '.join(replaced_tokens)

Step 3: Apply to Your DataFrame #

# Replace this with your actual DataFrame
df = pd.DataFrame({
    'body': [
        "David Beckham's dreams of kick starting his own...",
        "Ascension Island. Picture: NASA, via Wikicommo...",
        "",
        "HOUSTON - Wendy Davis continued to capitaliz...",
        "If something can't go on for ever, it won't. -...",
        "Published 04/10/2014 | 02:30 Taoiseach Enda...",
        "Ebola is having catastrophic economic conseque...",
        "A British man has been raped at the Oktoberfes..."
    ]
})

# Apply the function to the 'body' column
df['body'] = df['body'].apply(lambda x: replace_text_with_pos(x, pos_tags, tag_normalizer))

# Check the result
print(df['body'])

Option 2: Using spaCy (More Accurate, Scalable) #

If you need better POS tagging accuracy (especially for complex texts) or want to handle large datasets efficiently, spaCy is a great choice.

Step 1: Install & Import Dependencies #

pip install spacy
python -m spacy download en_core_web_sm

import pandas as pd
import spacy

# Load spaCy's pre-trained model
nlp = spacy.load('en_core_web_sm')

Step 2: Create a Replacement Function #

SpaCy uses broader, more intuitive POS tags (like NOUN, VERB, ADJ) out of the box, so no normalization is needed for most cases:

def replace_text_with_pos_spacy(text, pos_map):
    if pd.isna(text) or text.strip() == '':
        return text
    
    # Process text with spaCy
    doc = nlp(text)
    
    # Replace tokens using their POS tag
    replaced_tokens = [pos_map.get(token.pos_, token.text) for token in doc]
    
    return ' '.join(replaced_tokens)

Step 3: Apply to Your DataFrame (With Batch Processing for Speed) #

For large DataFrames, use spaCy's pipe method to process texts in batches:

# Batch process for better performance
processed_texts = []
for doc in nlp.pipe(df['body'].fillna(''), batch_size=50):
    replaced = ' '.join([pos_tags.get(token.pos_, token.text) for token in doc])
    processed_texts.append(replaced)

df['body'] = processed_texts

Key Tips #

• Edge Cases: The functions handle empty strings and NaN values to prevent runtime errors.
• Customization: Adjust the tag_normalizer (for NLTK) or pos_tags dict to match your exact tagging needs.
• Performance: SpaCy's batch processing is significantly faster than NLTK for large datasets.

内容的提问来源于stack exchange，提问作者Yikai