如何在PySpark DataFrame中将转义换行符\n替换为真实换行符

要解决这个问题，核心是识别并替换字符串中以两个字符形式存在的\n（即反斜杠+字母n），替换为实际的换行符。下面是两种可行的实现方式：

方法一：使用PySpark函数regexp_replace #

通过regexp_replace函数匹配转义的\n序列，替换为真实换行符：

from pyspark.sql import SparkSession
from pyspark.sql.functions import regexp_replace, col

# 初始化SparkSession（如果未初始化）
spark = SparkSession.builder.appName("ReplaceEscapedNewlines").getOrCreate()

# 你的原始DataFrame代码
columns = ["language","definition"]
data = [("Java", "Java is a high-level\\n class-based, object-oriented programming\\n language that is designed to have as few implementation\\n dependencies as possible"), ("Python", "Python is an interpreted\\n object-oriented, high-level\\n programming language")]
df = spark.createDataFrame(data).toDF(*columns)

# 替换转义换行符
df_processed = df.withColumn(
    "definition",
    regexp_replace(col("definition"), "\\\\n", "\n")
)

# 查看结果
df_processed.show(truncate=False)

说明 #

• 正则表达式\\\\n：在Python字符串中，每个反斜杠需要转义，因此\\\\n会被解析为正则中的\\n，恰好匹配原字符串中作为两个字符存在的\n。
• 替换值"\n"是Python中的真实换行符，会被Spark识别为换行。

方法二：使用Spark SQL语法 #

如果你更习惯SQL风格的操作，可以通过临时视图实现：

# 创建临时视图
df.createOrReplaceTempView("language_definitions")

# 执行SQL替换
df_processed = spark.sql("""
    SELECT 
        language, 
        regexp_replace(definition, '\\\\n', '\n') AS definition
    FROM language_definitions
""")

# 查看结果
df_processed.show(truncate=False)

处理后的输出效果 #

执行上述代码后，show的输出会符合你的预期：

+--------+----------------------------------------------------------------------------------------------------------------------------------------------------+
|language|definition                                                                                                                                          |
+--------+----------------------------------------------------------------------------------------------------------------------------------------------------+
|Java    |Java is a high-level
 class-based, object-oriented programming
 language that is designed to have as few implementation
 dependencies as possible|
|Python  |Python is an interpreted
 object-oriented, high-level
 programming language                                                                        |
+--------+----------------------------------------------------------------------------------------------------------------------------------------------------+

内容的提问来源于stack exchange，提问作者grinim