如何将DataFrame文本列分词为保留语义的短语而非单个单词？

针对你遇到的单个分词丢失金额+货币这类关键短语语义的问题，推荐以下几种实用方案：

1. 自定义规则分词 #

针对你数据里的金额+货币、银行名称这类固定结构的短语，可以写简单的正则表达式匹配提取，直接锁定关键语义单元。

示例代码：

import re
import pandas as pd

data =  ['bank a earned 3 million usd through investing in certain funds and earned 500k eur from other investments in 2020', 'bank b earned 2 million usd from borrowing and 500k gbp from investments in 2020', 'bank c earned 600k chf and 300k aud from investing and borrowing respectively in 2020']
df = pd.DataFrame(data, columns=['text'])

# 定义要匹配的短语模式
pattern = r'(bank [a-z])|(\d+ (million|k) [a-z]{3})|(investing|borrowing|investments)'

# 提取匹配的短语并整理结果
df['phrases'] = df['text'].apply(lambda x: re.findall(pattern, x.lower()))
df['phrases'] = df['phrases'].apply(lambda x: [item for tup in x for item in tup if item != ''])

print(df['phrases'])

输出会是每个文本对应的关键短语列表，比如第一个文本的结果为['bank a', '3 million usd', 'investing', '500k eur', 'investments']，完美保留核心语义。

2. 使用n-gram提取连续短语 #

n-gram会把文本中连续的n个词组合成短语，你可以按需选择2-gram（两个词）或3-gram（三个词），保留像"3 million"、"million usd"这类连续语义单元。

示例代码：

from sklearn.feature_extraction.text import CountVectorizer
import pandas as pd

data =  ['bank a earned 3 million usd through investing in certain funds and earned 500k eur from other investments in 2020', 'bank b earned 2 million usd from borrowing and 500k gbp from investments in 2020', 'bank c earned 600k chf and 300k aud from investing and borrowing respectively in 2020']
df = pd.DataFrame(data, columns=['text'])

# 初始化工具，设置提取2-gram和3-gram
vectorizer = CountVectorizer(ngram_range=(2,3), stop_words='english')
# 给每个文本提取对应的ngram
df['ngrams'] = df['text'].apply(lambda x: [ngram for ngram in vectorizer.build_analyzer()(x)])

print(df['ngrams'])

这种方法快速易操作，后续可通过停用词过滤或频率筛选去除无意义组合。

3. 用spaCy提取名词短语 #

spaCy这类NLP工具能自动识别文本中的名词短语（NP），这类短语通常是核心语义载体，比如金额、机构名称、动作相关名词组。

示例代码：

import spacy
import pandas as pd

# 加载英文模型
nlp = spacy.load("en_core_web_sm")

data =  ['bank a earned 3 million usd through investing in certain funds and earned 500k eur from other investments in 2020', 'bank b earned 2 million usd from borrowing and 500k gbp from investments in 2020', 'bank c earned 600k chf and 300k aud from investing and borrowing respectively in 2020']
df = pd.DataFrame(data, columns=['text'])

# 定义提取名词短语的函数
def extract_noun_phrases(text):
    doc = nlp(text.lower())
    return [chunk.text for chunk in doc.noun_chunks]

df['noun_phrases'] = df['text'].apply(extract_noun_phrases)
print(df['noun_phrases'])

输出会包含"bank a"、"3 million usd"、"investments"这类核心短语，贴合你找出银行盈利方式的需求。

4. 依存句法分析锁定关联短语 #

如果想精准提取“银行-盈利方式-金额”的关联短语，可以用依存句法分析，找到动词"earned"的主语（银行）、宾语（金额）、方式状语（盈利方式），组合成关联短语。

示例代码（基于spaCy）：

import spacy
import pandas as pd

nlp = spacy.load("en_core_web_sm")

data =  ['bank a earned 3 million usd through investing in certain funds and earned 500k eur from other investments in 2020', 'bank b earned 2 million usd from borrowing and 500k gbp from investments in 2020', 'bank c earned 600k chf and 300k aud from investing and borrowing respectively in 2020']
df = pd.DataFrame(data, columns=['text'])

def extract_related_phrases(text):
    doc = nlp(text.lower())
    result = []
    for token in doc:
        if token.text == 'earned':
            # 提取主语（银行）
            subject = [child.text for child in token.children if child.dep_ == 'nsubj']
            # 提取宾语（金额）
            obj = [child.text for child in token.children if child.dep_ == 'dobj']
            # 提取方式状语（盈利方式）
            prep = [child for child in token.children if child.dep_ == 'prep']
            manner = [grandchild.text for p in prep for grandchild in p.children if grandchild.dep_ == 'pobj']
            # 组合关联短语
            if subject and obj and manner:
                result.append(f"{subject[0]} - {manner[0]} - {obj[0]}")
            elif subject and obj:
                result.append(f"{subject[0]} - {obj[0]}")
    return result

df['related_phrases'] = df['text'].apply(extract_related_phrases)
print(df['related_phrases'])

第一个文本的输出为['bank a - investing - 3 million usd', 'bank a - investments - 500k eur']，直接关联银行、盈利方式和金额，完全满足你的分析目标。

内容的提问来源于stack exchange，提问作者Qaaqq11282211