如何高效在Pandas中拆分text列生成有序双词行？

要高效实现将text列拆分为每行两个连续单词并保留原有顺序的需求，可以利用Pandas的内置方法结合Python标准库工具，比手动列表推导式更简洁且性能更优。以下是两种高效方案：

方案一：利用explode+滚动窗口（适合大数据集） #

这种方法通过先展开单词再做滑动窗口拼接，避免Python层面循环，性能更出色：

import pandas as pd

df = pd.DataFrame({'col1':[22,23,44], 'col2': ['rr','gg','xx'], 'text': ['this is a sample text', 'this is another one','third example is a longer text']})

# 1. 将text列拆分为单词列表并展开成每行一个单词
df_expanded = df.assign(words=df['text'].str.split()).explode('words')

# 2. 按原数据行索引分组，生成连续两个单词的拼接结果
two_words_series = df_expanded.groupby(df_expanded.index)['words']\
                              .rolling(2)\
                              .agg(' '.join)\
                              .droplevel(0)\
                              .dropna()

# 3. 合并原数据的其他列与拆分结果
final_df = df.loc[two_words_series.index].assign(two_words=two_words_series.values)

最终final_df输出结果：

col1 col2                     text       two_words
0    22   rr    this is a sample text        this is
0    22   rr    this is a sample text          is a
0    22   rr    this is a sample text       a sample
0    22   rr    this is a sample text    sample text
1    23   gg       this is another one        this is
1    23   gg       this is another one     is another
1    23   gg       this is another one  another one
2    44   xx  third example is a longer text  third example
2    44   xx  third example is a longer text  example is
2    44   xx  third example is a longer text        is a
2    44   xx  third example is a longer text     a longer
2    44   xx  third example is a longer text  longer text

方案二：利用itertools.pairwise（简洁直观） #

如果使用Python 3.10+，可以借助itertools.pairwise快速生成连续单词对，代码更简洁：

import pandas as pd
import itertools

df = pd.DataFrame({'col1':[22,23,44], 'col2': ['rr','gg','xx'], 'text': ['this is a sample text', 'this is another one','third example is a longer text']})

# 生成每个文本的连续单词对，再展开为多行
final_df = df.assign(
    two_words=df['text'].str.split().apply(lambda x: list(itertools.pairwise(x)))
).explode('two_words').assign(
    two_words=lambda x: x['two_words'].str.join(' ')
)

该方案输出与方案一完全一致，pairwise是C实现的内置函数，效率比手动列表推导式更高。

性能说明 #

• 方案一依赖Pandas矢量化分组和滚动操作，在大数据集上优势明显；
• 方案二利用标准库高效工具，代码简洁且性能优异，适合大多数场景。

内容的提问来源于stack exchange，提问作者learner