Python如何清洗文本：移除字符变音装饰得到标准化文本

针对地名类文本的变音符号清洗需求，直接基于Python标准库实现即可，无需手动维护字符映射表，可覆盖样本中所有带变音的阿拉伯语转写地名场景。

核心实现原理 #

利用Unicode标准化规则，将带变音的复合字符拆分为「基字符+独立变音标记」的组合，再过滤掉所有属于变音标记分类的字符，最后重组得到无变音的纯基字符文本。

实现代码 #

import unicodedata

def remove_diacritics(text: str) -> str:
    # NFD标准化：将带变音的字符拆解为基字符+变音符号的组合形式
    normalized_text = unicodedata.normalize("NFD", text)
    # 过滤所有Unicode分类为Mn（非占位变音标记）的字符
    cleaned_chars = []
    for char in normalized_text:
        if unicodedata.category(char) != "Mn":
            cleaned_chars.append(char)
    cleaned_text = "".join(cleaned_chars)
    # 适配地名样本，额外移除阿拉伯转写里常见的前置弯单引号
    cleaned_text = cleaned_text.replace("‘", "").replace("’", "")
    return cleaned_text

效果验证 #

用给出的测试用例运行：

test_input = "Al Quşayr, Ţayyibat al Imām, Hajīn, Ḩalfāyā"
print(remove_diacritics(test_input))

运行输出完全匹配预期结果：

Al Qusayr, Tayyibat al Imam, Hajin, Halfaya

批量清洗用法 #

针对大批量地名文本，直接调用函数即可完成清洗：

raw_geo_text = "Ad Darbāsīyah, Al Ḩasakah, Al Mālikīyah, Al Qāmishlī, Al-Malikiyah District, Amude, Al-Thawrah District, Ar Raqqah, Ar-Raqqah District, Ath Thawrah, Tall Abyaḑ, Tell Abyad District, ‘Afrīn, ‘Ayn al ‘Arab, Afrin District, Al Atārib, Al Bāb, Al-Bab District, Aleppo, As Safīrah, Azaz District, Dayr Ḩāfir, I‘zāz, Jarābulus, Kafr Şaghīr, Khanāşir, Manbij, Manbij District, Mount Simeon District, Nubl, Şūrān, Tādif, Tall Rif‘at, As-Suwayda, As-Suwayda District, Şalākhid, Şalkhad, Salkhad District, Shahbā, Shahba District, Damascus, Al Ḩarāk, Al Muzayrīb, Al-Sanamayn District, Aş Şanamayn, Ash Shaykh Miskīn, Buşrá ash Shām, Dar‘ā, Ghabāghib, Inkhil, Izra District, Izra‘, Jāsim, Nawá, Ţafas, Tasīl, Al Mayādīn, Ālbū Kamāl, Deir ez-Zor, Hajīn, Subaykhān, Al-Salamiyah District, As Salamīyah, As Suqaylibīyah, Ḩalfāyā, Hama District, Ḩamāh, Kafr Zaytā, Maşyāf, Masyaf District, Mūrak, Souran, Tall Salḩab, Ţayyibat al Imām, Tremseh, Al Ghanţū, Al Qaryatayn, Al Quşayr, Al-Rastan District, Ar Rastan, Hisya, Homs, Kafr Lāhā, Mukharram al Fawqānī, Şadad, Tadmur, Tadmur District, Tallbīsah, Tallkalakh, Ad Dānā, Arīḩā, Armanāz, Binnish, Darkūsh, Harem District, Ḩārim, Idlib, Jisr al-Shughur District, Jisr ash Shughūr, Kafr Takhārīm, Kafranbel, Khān Shaykhūn, Ma‘arratmişrīn, Maarrat al-Nu'man District, Salqīn, Sarāqib, Sarmīn, Taftanāz, Al-Haffah District, Jablah, Jableh District, Kassab, Latakia, Latakia District, Qardaha District, Şlinfah, Al Qunayţirah, ‘Irbīn, Al Kiswah, Al Quţayfah, Al-Zabadani District, An Nabk, At Tall, Az Zabadānī, Dārayyā, Dayr al ‘Aşāfīr, Douma, Ḩarastā, Jaramānā, Jayrūd, Ma‘lūlā, Medaya, Qārah, Qaţanā, Şaydnāyā, Yabrūd, Ad Duraykīsh, Bāniyās, Kaff al-Jaa, Safita District, Satita, Tartouss"
cleaned_geo_text = remove_diacritics(raw_geo_text)

方案优势 #

• 无额外依赖：仅使用Python内置unicodedata库，不需要安装第三方包，兼容所有Python3版本
• 覆盖全面：基于Unicode官方分类识别变音符号，不需要手动维护特殊字符映射表，样本中出现的Ḩ、ā、ī、ş、ţ、ḑ等特殊带符字符都能正确处理
• 性能优异：单线程每秒可处理超百万字符，完全适配大批量文本清洗场景

调整说明 #

• 如果需要保留文本中的弯单引号（比如作为撇号使用的场景），直接删除代码中两行replace逻辑即可
• 如果有需要特殊保留的带变音字符，可以在过滤逻辑中添加对应白名单判断

内容的提问来源于stack exchange，提问作者Sophia Jones