如何在Pandas DataFrame中过滤非法字符？代码问题排查与修正

我写了一段代码想清理邮件列表里的非法字符，用bad_chars()函数处理Pandas DataFrame中的邮箱数据，但执行后输出全是字符“ü”，根本没法正确过滤非法字符。请问问题出在哪？该怎么实现需求？

我的代码 #

import pandas as pd
import numpy as np


excelRead = pd.read_excel('mailing.xlsx')
excelRead.dropna(inplace= True)
badCharsList = ["ü", "ı", "ö", "ç", "ş", "ğ", "!", "#", "$", "%", "&", "'", "*", "+",  "/", "=", "?" "^", "`", "{", "|", "}", "~", "(",")",",",":",";","<",">","[", " "]

def bad_chars(x):
    for i in badCharsList:
        if i.lower() not in x.lower():
            return i
        else:
            return np.nan

excelTest = excelRead[excelRead['mails'].str.endswith("@gmail.com", na=False) | excelRead['mails'].str.endswith("@hotmail.com", na=False) | excelRead['mails'].str.endswith("@outlook.com", na=False) | excelRead['mails'].str.endswith("@icloud.com", na=False) | excelRead['mails'].str.endswith("@windowslive.com", na=False) | excelRead['mails'].str.endswith("@yandex.com", na=False) | excelRead['mails'].str.endswith("@mynet.com", na=False) | excelRead['mails'].str.endswith("@hotmail.com.tr", na=False) | excelRead['mails'].str.endswith("@yahoo.com", na=False)]

lower = excelTest['mails'].str.lower()
testBad = excelTest['mails'].apply(bad_chars)

print(testBad)

执行输出 #

0         ü
1         ü
2         ü
3         ü
4         ü
         ..
107808    ü
107809    ü
107810    ü
107811    ü
107812    ü
Name: mails, Length: 104507, dtype: object

原始邮箱数据示例 #

0            okanmercannn@hotmail.com
1         06hvm42hotmailcom@gmail.com
2              adanasenol01@gmail.com
3          sezersenturk6305@gmail.com
4                alyasu1903@gmail.com
                     ...
107808        elifyucel2566@gmail.com
107809        yayla19871987@gmail.com
107810         zeynepyilkus@gmail.com
107811    pathoss_theodra@hotmail.com
107812           ziver.7340@gmail.com

1. 函数逻辑完全错误：bad_chars()里的for循环只执行第一次迭代就直接return，根本没遍历完所有非法字符。而且逻辑搞反了——你现在是检查第一个非法字符ü是否不在邮箱里，只要不在就返回ü；你的测试邮箱都不含ü，所以全返回ü。
2. 列表语法错误：badCharsList里的"?"和"^"之间少了逗号，会被识别成一个字符串"?^"，导致这两个字符无法被单独匹配。

方案1：标记包含非法字符的邮箱 #

如果需求是找出哪些邮箱存在非法字符，修改函数逻辑：

# 先修复列表的逗号问题
badCharsList = ["ü", "ı", "ö", "ç", "ş", "ğ", "!", "#", "$", "%", "&", "'", "*", "+",  "/", "=", "?", "^", "`", "{", "|", "}", "~", "(",")",",",":",";","<",">","[", " "]

def has_bad_chars(x):
    x_lower = x.lower()
    # 遍历所有非法字符，只要有一个存在就返回True
    for char in badCharsList:
        if char.lower() in x_lower:
            return True
    # 遍历完都没找到，返回False
    return False

# 应用函数得到标记结果
testBad = excelTest['mails'].apply(has_bad_chars)
print(testBad)

方案2：直接清理邮箱中的非法字符 #

如果需求是移除邮箱里的非法字符，用正则表达式结合str.replace更高效：

import re

# 修复后的非法字符列表
badCharsList = ["ü", "ı", "ö", "ç", "ş", "ğ", "!", "#", "$", "%", "&", "'", "*", "+",  "/", "=", "?", "^", "`", "{", "|", "}", "~", "(",")",",",":",";","<",">","[", " "]

# 转成正则匹配模式，避免特殊字符转义问题
bad_chars_pattern = '[' + re.escape(''.join(badCharsList)) + ']'

# 直接替换所有非法字符为空字符串
cleaned_mails = excelTest['mails'].str.replace(bad_chars_pattern, '', regex=True)
print(cleaned_mails)

额外优化：简化邮箱后缀判断 #

你原来的后缀判断代码太冗长，可以简化成：

allowed_domains = [
    "@gmail.com", "@hotmail.com", "@outlook.com", "@icloud.com",
    "@windowslive.com", "@yandex.com", "@mynet.com", 
    "@hotmail.com.tr", "@yahoo.com"
]
# 用tuple作为endswith的参数，一次判断所有后缀
excelTest = excelRead[excelRead['mails'].str.endswith(tuple(allowed_domains), na=False)]

内容的提问来源于stack exchange，提问作者Ozans