如何识别并移除语料库中含数字的术语与纯数字？

Got it, let's work through this. You want to keep only pure words in your text, stripping out any numbers or terms that include numbers (like Alice2B, 7/2, 25.1, etc.). Here's a straightforward way to do this with regular expressions—this works well in Python, which is widely used for text processing tasks like this.

Step-by-Step Explanation #

• Regex Pattern: We'll use a regex that targets any "term" (sequence of non-whitespace characters) containing at least one digit. The pattern \b[\w/–-]*\d[\w/–-]*\b covers:
  • \b: Word boundary to ensure we match full terms, not parts of words
  • [\w/–-]*: Matches any combination of letters, digits, underscores, slashes, en dashes, or hyphens (adjust this set if your text has other special characters)
  • \d: Ensures the term contains at least one digit
• Replace & Clean: Replace all matches with empty strings, then clean up extra spaces left from the removals.

Code Example #

import re

# Your input corpus
original_text = "Alice2B Visum 7/2 Dann 394–3973-3 ging sie nach Hollywood dort als 25.1 Drehbuchautorin arbeiten Kurz 2006 nach ihrer 329–49 Ankunft lernte sie den Filmregisseur 02/ayn Cecil"

# Remove all terms containing numbers (including those with special symbols like /, –)
cleaned_text = re.sub(r'\b[\w/–-]*\d[\w/–-]*\b', '', original_text)

# Remove extra spaces created by the replacements
cleaned_text = re.sub(r'\s+', ' ', cleaned_text).strip()

print(cleaned_text)

Output #

Visum Dann ging sie nach Hollywood dort als Drehbuchautorin arbeiten Kurz nach ihrer Ankunft lernte sie den Filmregisseur Cecil

This will exactly give you the result you're looking for. If your corpus has other special characters tied to numbers, just add them to the character set [\w/–-] in the regex pattern to ensure those terms are also caught.

内容的提问来源于stack exchange，提问作者SahelSoft