Qdrant中count结果与upsert插入行数不符问题咨询

我按如下方式初始化Qdrant集合：

client = QdrantClient(location=":memory:")
my_collection = "my_collection"
client.delete_collection(my_collection)
if not client.collection_exists(my_collection):
  client.create_collection(
      collection_name=my_collection,
      vectors_config=models.VectorParams(size=768, distance=models.Distance.COSINE)
  )

并通过以下方法将HuggingFace数据集插入集合：

def insert_dataset_to_qdrant(dataset_to_process, client):
  np.save("vectors", np.array(dataset_to_process['embeddings']), allow_pickle=False)
  ids = list(range(dataset_to_process.num_rows))
  embeddings = np.load("vectors.npy").tolist()
  payload = dataset_to_process.select_columns([
      'text', 'postcard_id'
  ]).to_pandas().to_dict(orient="records")

  batch_size = 1000

  for i in range(0, dataset_to_process.num_rows, batch_size):
      low_idx = min(i+batch_size, dataset_to_process.num_rows)

      batch_of_ids = ids[i: low_idx]
      batch_of_embs = embeddings[i: low_idx]
      batch_of_payloads = payload[i: low_idx]

      client.upsert(
          collection_name=my_collection,
          points=models.Batch(
              ids=batch_of_ids,
              vectors=batch_of_embs,
              payloads=batch_of_payloads
          )
      )

随后我插入了多个数据集：

dataset1.shape (9778, 5)
dataset2.shape (9678, 4)
dataset3.shape (6118, 4)
dataset4.shape (14314, 4)
dataset5.shape (12084, 4)
dataset6.shape (6202, 4)
dataset7.shape (18994, 4)
dataset8.shape (10760, 4)

但执行以下代码：

client.count(
    collection_name=my_collection,
    exact=True
)

得到结果：CountResult(count=18994)。我认为count结果应为各数据集行数总和，请问为何出现此情况？

核心原因 #

你使用的upsert操作逻辑是存在则更新、不存在则插入，但每个数据集插入时生成的ids都是从0开始的连续整数（list(range(dataset_to_process.num_rows))）。后续插入的数据集会直接覆盖掉前面相同ID的点，最终只有最后插入的数据集（dataset7，行数18994）的内容被完整保留，所以count结果等于该数据集的行数。

解决办法 #

需要为所有数据集的点分配全局唯一的ID，避免重复覆盖，以下是两种可行方案：

1. 累计递增ID
   维护一个全局ID计数器，每次插入数据集时从当前计数器值开始生成ID，插入完成后更新计数器：

# 初始化全局ID计数器
global_id_counter = 0

def insert_dataset_to_qdrant(dataset_to_process, client):
    global global_id_counter
    np.save("vectors", np.array(dataset_to_process['embeddings']), allow_pickle=False)
    # 生成从当前计数器开始的连续ID
    ids = list(range(global_id_counter, global_id_counter + dataset_to_process.num_rows))
    embeddings = np.load("vectors.npy").tolist()
    payload = dataset_to_process.select_columns([
        'text', 'postcard_id'
    ]).to_pandas().to_dict(orient="records")

    batch_size = 1000

    for i in range(0, dataset_to_process.num_rows, batch_size):
        low_idx = min(i+batch_size, dataset_to_process.num_rows)

        batch_of_ids = ids[i: low_idx]
        batch_of_embs = embeddings[i: low_idx]
        batch_of_payloads = payload[i: low_idx]

        client.upsert(
            collection_name=my_collection,
            points=models.Batch(
                ids=batch_of_ids,
                vectors=batch_of_embs,
                payloads=batch_of_payloads
            )
        )
    # 更新全局ID计数器
    global_id_counter += dataset_to_process.num_rows

2. 使用UUID作为唯一ID
   直接为每个点生成UUID作为ID，确保全局唯一性：

import uuid

def insert_dataset_to_qdrant(dataset_to_process, client):
    np.save("vectors", np.array(dataset_to_process['embeddings']), allow_pickle=False)
    # 为每个点生成UUID
    ids = [str(uuid.uuid4()) for _ in range(dataset_to_process.num_rows)]
    embeddings = np.load("vectors.npy").tolist()
    payload = dataset_to_process.select_columns([
        'text', 'postcard_id'
    ]).to_pandas().to_dict(orient="records")

    batch_size = 1000

    for i in range(0, dataset_to_process.num_rows, batch_size):
        low_idx = min(i+batch_size, dataset_to_process.num_rows)

        batch_of_ids = ids[i: low_idx]
        batch_of_embs = embeddings[i: low_idx]
        batch_of_payloads = payload[i: low_idx]

        client.upsert(
            collection_name=my_collection,
            points=models.Batch(
                ids=batch_of_ids,
                vectors=batch_of_embs,
                payloads=batch_of_payloads
            )
        )

内容的提问来源于stack exchange，提问作者gisly