np.setdiff1d报TypeError：结构化数组无共同dtype的原因及解决

问题重现 #

执行以下代码时，调用np.setdiff1d触发TypeError：

import pandas as pd
import numpy as np

data1 = {'a' : [32,156], 'b' :[56,177]}
data2 = {'c' : [12,32,12,45,32,45], 'd' :[11,56,76,43,44,45], 'e': [111,156,176,143,144,145], 'f':[411,456,476,443,444,445] }

df1 = pd.DataFrame(data1)
df2 = pd.DataFrame(data2)

# 转为结构化数组
npdf1= df1.to_records(index=False)
npdf2= df2.to_records(index=False)

# 首次报错
diff = np.setdiff1d(npdf1,npdf2[['c','e']])

# 尝试转普通数组后仍报错
npdf1 = np.array(npdf1)
df2a = df2[['c','e']]
npdf2a = df2a.to_records(index=False)
npdf2a = np.array(npdf2a)
diff = np.setdiff1d(npdf1,npdf2a)

报错信息：

TypeError: Cannot compare structured arrays unless they have a common dtype. I.e. np.result_type(arr1, arr2) must be defined.

报错原因 #

• 结构化数组的dtype是完整的元数据集合，不仅包含元素类型（如<i8），还包含字段名、字节偏移、元素大小等信息。你的两个数组字段名分别为('a','b')和('c','e')，numpy判定为完全不同的dtype，无法生成兼容的共同类型，因此无法比较。
• 直接用np.array()转换结构化数组，只是将recarray转为普通ndarray，但结构化dtype并未改变，依然存在字段名不匹配的问题，所以报错依旧。

解决方法 #

方法1：转为普通二维数值数组 #

将结构化数组转换为统一元素类型的普通二维数组，消除结构化dtype的差异：

import pandas as pd
import numpy as np

data1 = {'a' : [32,156], 'b' :[56,177]}
data2 = {'c' : [12,32,12,45,32,45], 'e': [111,156,176,143,144,145]}

df1 = pd.DataFrame(data1)
df2 = pd.DataFrame(data2)

# 从DataFrame直接提取数值数组，或转换结构化数组为普通数组
npdf1 = df1.values  # 等价于 npdf1.view('i8').reshape(-1,2)
npdf2a = df2[['c','e']].values

diff = np.setdiff1d(npdf1, npdf2a)
print(diff)
# 输出: [ 32  56 156 177]

方法2：统一结构化数组的dtype #

重新定义两个数组的dtype，让字段名完全一致，这样numpy就能识别为兼容类型：

import pandas as pd
import numpy as np

data1 = {'a' : [32,156], 'b' :[56,177]}
data2 = {'c' : [12,32,12,45,32,45], 'e': [111,156,176,143,144,145]}

df1 = pd.DataFrame(data1)
df2 = pd.DataFrame(data2)

npdf1 = df1.to_records(index=False).astype([('x', 'i8'), ('y', 'i8')])
npdf2a = df2[['c','e']].to_records(index=False).astype([('x', 'i8'), ('y', 'i8')])

diff = np.setdiff1d(npdf1, npdf2a)
print(diff)
# 输出: rec.array([( 32,  56), (156, 177)], dtype=[('x', '<i8'), ('y', '<i8')])

方法3：用pandas集合操作替代 #

利用Python集合的差集特性，结合pandas数据提取，更直观简洁：

import pandas as pd
import numpy as np

data1 = {'a' : [32,156], 'b' :[56,177]}
data2 = {'c' : [12,32,12,45,32,45], 'e': [111,156,176,143,144,145]}

df1 = pd.DataFrame(data1)
df2 = pd.DataFrame(data2)

# 将每行转为元组，存入集合
set1 = set(tuple(row) for row in df1.values)
set2 = set(tuple(row) for row in df2[['c','e']].values)

# 求差集并转为数组
diff = np.array(list(set1 - set2))
print(diff)
# 输出: [[ 32  56]
#        [156 177]]

内容的提问来源于stack exchange，提问作者user1412