PySpark合并同主键DataFrame并标注列来源的实现咨询

该需求完全可以实现，以下是两种常用数据处理框架下的操作方法：

Pandas 实现 #

Pandas的合并功能原生支持区分同名列来源，两种常用写法如下：

1. 提前重命名非主键列后合并（更灵活，支持自定义前缀格式）

import pandas as pd

# 构造样例df1、df2
df1 = pd.DataFrame([
    {"ID": "401148EE-9BA6-4BAA-B113-ED694B0F5BED", "CURRENCY": 100.00},
    {"ID": "E90ED21E-C60F-412C-8305-DB5675DA7A5E", "CURRENCY": 1000.00}
])
df2 = pd.DataFrame([
    {"ID": "401148EE-9BA6-4BAA-B113-ED694B0F5BED", "CURRENCY": 200.00},
    {"ID": "E90ED21E-C60F-412C-8305-DB5675DA7A5E", "CURRENCY": 2000.00}
])

# 批量给非主键列添加来源前缀
df1_rename = df1.rename(columns={col: f"DF1.{col}" for col in df1.columns if col != "ID"})
df2_rename = df2.rename(columns={col: f"DF2.{col}" for col in df2.columns if col != "ID"})

# 按主键ID合并
result = df1_rename.merge(df2_rename, on="ID", how="inner")

2. 使用merge自带的suffixes参数处理

result = df1.merge(df2, on="ID", suffixes=(".DF1", ".DF2"))
# 如果需要前缀格式可后续调整列名：result.columns = result.columns.str.replace(r'^(.*)\.DF1$', r'DF1.\1', regex=True)

PySpark 实现 #

如果是Spark场景，操作逻辑和Pandas一致，示例代码如下：

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("df_merge").getOrCreate()

# 构造样例df1、df2
df1 = spark.createDataFrame([
    ("401148EE-9BA6-4BAA-B113-ED694B0F5BED", 100.00),
    ("E90ED21E-C60F-412C-8305-DB5675DA7A5E", 1000.00)
], schema=["ID", "CURRENCY"])
df2 = spark.createDataFrame([
    ("401148EE-9BA6-4BAA-B113-ED694B0F5BED", 200.00),
    ("E90ED21E-C60F-412C-8305-DB5675DA7A5E", 2000.00)
], schema=["ID", "CURRENCY"])

# 重命名非主键列后合并
df1_rename = df1.withColumnRenamed("CURRENCY", "DF1.CURRENCY")
df2_rename = df2.withColumnRenamed("CURRENCY", "DF2.CURRENCY")
result = df1_rename.join(df2_rename, on="ID", how="inner")

以上两种方法输出的结果都和你要求的格式完全一致，如果存在多个非主键重名列，也可以批量处理无需逐个修改列名。

内容的提问来源于stack exchange，提问作者Bruno