如何使用哈希函数为含NA的数据框生成唯一观测ID

针对你需要给含120个变量的数据集生成唯一ID、且不想手动指定所有列的需求，以下是几种实用方案，同时处理数据中的NA问题：

方法1：用apply+哈希生成唯一ID #

无需指定列名，自动遍历所有列拼接后生成哈希，同时处理缺失值避免冲突：

library(digest)

# 加载示例数据
df <- structure(list(Class = structure(c(1L, 2L, 3L, 4L, 1L, 2L, 3L, 
4L, 1L, 2L, 3L, 4L, 1L, 2L, 3L, 4L, 1L, 2L, 3L, 4L, 1L, 2L, 3L, 
4L, 1L, 2L, 3L, 4L, 1L, 2L, 3L, 4L), levels = c("1st", "2nd", 
"3rd", "Crew"), class = "factor"), Sex = structure(c(1L, 1L, 
1L, 1L, 2L, 2L, 2L, 2L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 1L, 1L, 
1L, 1L, 2L, 2L, 2L, 2L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 2L), levels = c("Male", 
"Female"), class = "factor"), Age = structure(c(1L, NA, 1L, NA, 
1L, NA, 1L, 1L, 2L, 2L, NA, 2L, 2L, 2L, 2L, NA, 1L, 1L, 1L, NA, 
NA, 1L, 1L, 1L, NA, 2L, 2L, 2L, 2L, 2L, 2L, NA), levels = c("Child", 
"Adult"), class = "factor"), Survived = structure(c(1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L), levels = c("No", 
"Yes"), class = "factor"), Freq = c(0, 0, 35, 0, 0, 0, 17, 0, 
118, 154, 387, 670, 4, 13, 89, 3, 5, 11, 13, 0, 1, 13, 14, 0, 
57, 14, 75, 192, 140, 80, 76, 20)), row.names = c(NA, -32L), class = "data.frame")

# 生成哈希ID：将NA替换为专属标记避免与字符串"NA"混淆
df$obs_id <- apply(df, 1, function(row) {
  row[is.na(row)] <- "__NA__"
  digest(paste(row, collapse = "|"), algo = "md5")
})

方法2：用dplyr+unite简化操作 #

利用dplyr的everything()自动选中所有列，合并后生成哈希：

library(dplyr)
library(digest)

df <- df %>%
  unite(col = temp_col, everything(), sep = "|", na.rm = FALSE) %>%
  mutate(temp_col = gsub("NA", "__NA__", temp_col)) %>% # 替换缺失值标记
  mutate(obs_id = sapply(temp_col, digest, algo = "md5")) %>%
  select(-temp_col)

方法3：直接用行索引（仅适用于无重复行的场景） #

如果数据框每行都是唯一观测、无完全重复的行，直接用行号作为ID最简便：

df$obs_id <- seq(nrow(df))

额外优化：大数据集用data.table提升速度 #

如果数据集行数较多（如百万级），用data.table的向量化操作效率更高：

library(data.table)
library(digest)

setDT(df)
df[, temp_col := do.call(paste, c(.SD, sep = "|", na.string = "__NA__"))]
df[, obs_id := sapply(temp_col, digest, algo = "md5")]
df[, temp_col := NULL]

注意事项 #

• 缺失值处理：必须将NA替换为专属标记（如__NA__），避免与数据中真实的"NA"字符串产生哈希冲突。
• 哈希算法：默认用md5生成短且唯一的ID，也可通过修改digest的algo参数使用sha1等其他算法。

内容的提问来源于stack exchange，提问作者ECII