如何在MongoDB Realm中删除重复数据？distinct方法不生效如何解决

你现有代码的核心错误如下：

• collection("glassdoordbs").distinct() 调用错误：distinct()方法必须传入要去重的字段名作为参数，返回的是该字段所有不重复值的数组，不是集合实例，所以后续你调用.find()/.count()都会直接报错。
• dropDups是MongoDB旧版本创建唯一索引时的可选参数，3.0+版本已经废弃，不能用来做运行时去重或者删除重复数据。

方案1：查询时临时对结果去重（不修改底层集合数据） #

用聚合管道的$group阶段实现按指定字段去重，同时支持分页、查询条件匹配，修改后的代码如下：

const {jobsPerPage = 100, page = 0} = payload.query;

let query = {};
if (payload.query.jobPosition) {
  query = { $text: { $search: payload.query.jobPosition } }
} else if (payload.query.location) {
  query = { "location": { $eq: payload.query.location } }
} else if (payload.query.companyName) {
  query = { $text: { $search: payload.query.companyName } }
} else if (payload.query.jobDescription) {
  query = { $text: { $search: payload.query.jobDescription } }
}
  
const collection = context.services.get("mongodb-atlas").db("sample_jobs").collection("glassdoordbs");
// 示例按职位+公司名+地点三个组合字段去重，可根据实际业务调整去重依据的字段
const aggPipeline = [
  { $match: query },
  // 按去重字段分组，保留第一条匹配数据
  { $group: {
    _id: { jobPosition: "$jobPosition", companyName: "$companyName", location: "$location" },
    doc: { $first: "$$ROOT" }
  }},
  // 还原标准文档结构
  { $replaceRoot: { newRoot: "$doc" } },
  // 分页逻辑
  { $skip: page * jobsPerPage },
  { $limit: jobsPerPage }
];
// 统计去重后的总条数
const countPipeline = [
  { $match: query },
  { $group: {
    _id: { jobPosition: "$jobPosition", companyName: "$companyName", location: "$location" }
  }},
  { $count: "total" }
]
let jobsList = await collection.aggregate(aggPipeline).toArray();
const countRes = await collection.aggregate(countPipeline).toArray();
const allResults = countRes[0]?.total || 0;

jobsList.forEach(job => {
  job._id = job._id.toString();
});

const responseData = {
  jobs: jobsList,
  page: page.toString(),
  filters: {},
  entries_per_page: jobsPerPage.toString(),
  total_results: allResults.toString(),
  totalPages: Math.ceil(allResults / jobsPerPage).toString()
};

return responseData;

方案2：永久删除集合内的重复数据 #

如果需要直接清理集合里的重复记录，可运行以下一次性脚本，执行后建议添加唯一索引避免后续产生重复数据：

const collection = context.services.get("mongodb-atlas").db("sample_jobs").collection("glassdoordbs");
// 按去重字段分组，保留_id最小的记录，删除其余重复项
const delPipeline = [
  { $group: {
    _id: { jobPosition: "$jobPosition", companyName: "$companyName", location: "$location" },
    keepId: { $first: "$_id" },
    duplicateIds: { $addToSet: "$_id" }
  }},
  // 过滤出存在重复的分组
  { $project: {
    _id: 0,
    deleteIds: { $setDifference: ["$duplicateIds", ["$keepId"]] }
  }},
  { $unwind: "$deleteIds" }
]
const toDelete = await collection.aggregate(delPipeline).map(item => item.deleteIds).toArray();
if (toDelete.length > 0) {
  await collection.deleteMany({ _id: { $in: toDelete } })
}
// 创建唯一索引避免后续重复数据写入
await collection.createIndex(
  { jobPosition: 1, companyName: 1, location: 1 },
  { unique: true }
)

内容的提问来源于stack exchange，提问作者Christopher