Python标点移除脚本未删除引号？求助问题排查

问题背景 #

我在学习Python，编写了一个移除文本所有标点的脚本，用于后续词频统计和词云生成。但将脚本应用到《傲慢与偏见》的文本片段时，尽管已经把引号加入punctuation变量，文本中的弯引号却始终没有被移除。

我的代码如下：

text = '''      “Already arisen?” repeated Mr. Bennet. “What, has she frightened
      away some of your lovers? Poor little Lizzy! But do not be cast
      down. Such squeamish youths as cannot bear to be connected with a
      little absurdity are not worth a regret. Come, let me see the
      list of pitiful fellows who have been kept aloof by Lydia’s
      folly.”'''
punctuation = '''!()-[]{};:'"”,<>./?@#$%^&*_~'''
text = text.lower()
clean = ""

for letter in text:
    if letter not in punctuation:
        clean += letter
print(clean)

运行后文本转为小写，但弯引号未被移除，结果如下：

“already arisen” repeated mr bennet “what has she frightened
      away some of your lovers poor little lizzy but do not be cast
      down such squeamish youths as cannot bear to be connected with a
      little absurdity are not worth a regret come let me see the
      list of pitiful fellows who have been kept aloof by lydia’s
      folly.”

原因分析 #

你添加到punctuation里的是ASCII直引号（"），但文本中的引号是Unicode弯引号（“”），这是两种完全不同的字符，因此脚本无法识别并移除弯引号。另外文本中的撇号（’）也是Unicode字符，同样不在默认的标点列表里。

解决方法 #

直接将弯引号（“”）和撇号（’）添加到punctuation字符串中即可：

修改后的代码

text = '''      “Already arisen?” repeated Mr. Bennet. “What, has she frightened
      away some of your lovers? Poor little Lizzy! But do not be cast
      down. Such squeamish youths as cannot bear to be connected with a
      little absurdity are not worth a regret. Come, let me see the
      list of pitiful fellows who have been kept aloof by Lydia’s
      folly.”'''
# 加入弯引号和撇号
punctuation = '''!()-[]{};:'"”,<>./?@#$%^&*_~“”’'''
text = text.lower()
clean = ""

for letter in text:
    if letter not in punctuation:
        clean += letter
print(clean)

运行结果

already arisen repeated mr bennet what has she frightened
      away some of your lovers poor little lizzy but do not be cast
      down such squeamish youths as cannot bear to be connected with a
      little absurdity are not worth a regret come let me see the
      list of pitiful fellows who have been kept aloof by lydias
      folly

更高效的写法（可选）

可以利用Python标准库string.punctuation来获取所有ASCII标点，再手动补充Unicode弯引号和撇号，同时用列表推导式简化代码：

import string

text = '''      “Already arisen?” repeated Mr. Bennet. “What, has she frightened
      away some of your lovers? Poor little Lizzy! But do not be cast
      down. Such squeamish youths as cannot bear to be connected with a
      little absurdity are not worth a regret. Come, let me see the
      list of pitiful fellows who have been kept aloof by Lydia’s
      folly.”'''
# 合并ASCII标点与Unicode特殊标点
punctuation = string.punctuation + "“”’"
text = text.lower()
clean = ''.join([char for char in text if char not in punctuation])
print(clean)

内容的提问来源于stack exchange，提问作者oedma621