用户活跃后Google Cloud Function的Firestore查询变慢问题排查

问题背景 #

我有一个用于查询Firestore的Google Cloud Function（GCF），同时部署了供用户读写Firestore的应用。用户未活跃时GCF运行正常，但用户活跃数分钟后，GCF中的Firestore操作开始出现严重延迟——比如简单的文档更新耗时长达27秒，且用户停止活跃后延迟仍无法恢复。

相关代码 #

exports.processDocument = functions
  .runWith({ timeoutSeconds: 120 })
  .https.onRequest(async (req, res) => {
    const { documentId } = req.body;
    const id = req.body.documentId;

    if (!documentId) {
      return res.status(400).send("Missing documentId");
    }

    // Send a 200 response immediately
    res.status(200).send("Processing document...");

    const db = admin.firestore();
    const oneMindDocRef = db.collection("OneMind").doc(documentId);

    console.log(id, `Setting ${oneMindDocRef.id} document to being processed.`);
    await oneMindDocRef.update({
      processing: true,
      votingUsers: [],
    });
    console.log(id, `Set ${oneMindDocRef.id} document to being processed.`);

    const oneMindDoc = await oneMindDocRef.get();
    console.log(id, "Got OneMind document data");

    const activeUsers = oneMindDoc.data().activeUsers;
    const roundNumber = oneMindDoc.data().roundNumber;
    console.log(id, "Got activeUsers and roundNumber");

    if (!activeUsers || activeUsers.length === 0) {
      console.log(
        id,
        `No active users. Resetting ${oneMindDocRef.id} document.`
      );
      await oneMindDocRef.update({
        lastUpdated: admin.firestore.FieldValue.serverTimestamp(),
        processing: false,
        activeUsersCount: 0,
        activeUsers: [],
      });
      console.log(id, `${oneMindDocRef.id} document reset.`);

      console.log(`Document ${oneMindDocRef.id} reset due to no active users.`);
      return;
    }
    const proposedMessagesRef = db.collection("proposedMessages");

    if (oneMindDoc.data().userFeedbackMechanism == "slider") {
      await calculateWeightedAverageScores(db, documentId);
    }

    const proposedMessagesDocs = await proposedMessagesRef
      .where("roundNumber", "==", roundNumber)
      .where("chatroomName", "==", oneMindDoc.data().chatroomName)
      .orderBy("votes", "desc")
      .orderBy("creation", "asc")
      .get();

    console.log(id, "Got all proposed messages for the current round.");
    if (proposedMessagesDocs.docs.length === 0) {
      await oneMindDocRef.update({
        activeUsers: [],
        activeUsersCount: 0,
        lastUpdated: admin.firestore.FieldValue.serverTimestamp(),
        processing: false,
      });
      console.log(
        id,
        `${oneMindDocRef.id} document reset due to no proposed messages.`
      );

      console.log(
        `Document ${oneMindDocRef.id} reset due to no proposed messages.`
      );
      return;
    }

    const highestVotedMessageDoc = proposedMessagesDocs.docs[0];
    const highestVotedMessageData = highestVotedMessageDoc.data();

    console.log(id, 'Adding winning message to "messages" collection');
    const {
      message,
      creation,
      translatedText,
      creator,
      imageUrl,
      chatroomName,
      speaker,
    } = highestVotedMessageData;

    const messageRef = db.collection("messages").doc();
    const messageData = {
      creation: creation,
      creator: creator,
      speaker: speaker,
      roundNumber: roundNumber,
    };
    console.log("New message doc id: ", messageRef.id);

    if (imageUrl) {
      messageData.imageUrl = imageUrl;
    }

    if (message) {
      messageData.message = message;
    }

    if (translatedText) {
      messageData.translatedText = translatedText;
    }

    if (chatroomName) {
      messageData.chatroomName = chatroomName;
    }

    await messageRef.set(messageData);
    console.log(id, "New document created in messages collection.");

    let newCurrentSpeaker = oneMindDoc.data().currentSpeaker;
    if (oneMindDoc.data().chatroomName !== "OneMind") {
      newCurrentSpeaker =
        newCurrentSpeaker === oneMindDoc.data().team1
          ? oneMindDoc.data().team2
          : oneMindDoc.data().team1;
      console.log(
        `Updating current speaker from ${oneMindDoc.data().currentSpeaker} to ${newCurrentSpeaker}`
      );
    }

    console.log(id, `Resetting ${oneMindDocRef.id} document for next message`);
    await oneMindDocRef.update({
      activeUsers: [],
      activeUsersCount: 0,
      lastUpdated: admin.firestore.FieldValue.serverTimestamp(),
      processing: false,
      currentSpeaker: newCurrentSpeaker,
      roundNumber: admin.firestore.FieldValue.increment(1),
    });
    console.log(id, `${oneMindDocRef.id} document reset.`);
  });

延迟相关日志 #

2024-10-10 18:31:05.099 EDT processDocument4873x1mfdaij Function execution started
2024-10-10 18:31:05.256 EDT processDocument4873x1mfdaij about to start waiting
2024-10-10 18:31:05.257 EDT processDocument4873x1mfdaij done waiting
2024-10-10 18:31:05.257 EDT processDocument4873x1mfdaij Function execution took 157 ms, finished with status code: 200
2024-10-10 18:31:05.257 EDT processDocument4873x1mfdaij OneMind Setting OneMind document to being processed.
2024-10-10 18:31:07.955 EDT processDocument4873x1mfdaij OneMind Set OneMind document to being processed.
2024-10-10 18:31:34.255 EDT processDocument4873x1mfdaij OneMind Got OneMind document data
2024-10-10 18:31:41.855 EDT processDocument4873x1mfdaij OneMind Got activeUsers and roundNumber
2024-10-10 18:31:41.855 EDT processDocument4873x1mfdaij OneMind No active users. Resetting OneMind document.
2024-10-10 18:31:51.860 EDT processDocument4873x1mfdaij OneMind OneMind document reset.
2024-10-10 18:31:51.860 EDT processDocument4873x1mfdaij Document OneMind reset due to no active users.

注：日志中最后两行间隔27秒，期间仅执行Firestore文档更新操作

核心原因分析 #

1. 单文档并发锁竞争 #

Firestore对单个文档的写操作会施加排他锁，同一时间仅允许一个写操作执行。当用户活跃时，应用会频繁更新OneMind文档的activeUsers字段，同时GCF也会多次更新该文档（标记processing、重置字段等），大量并发请求会导致锁排队。这种排队会累积，即使用户停止活跃，之前积压的锁等待请求仍需要时间处理，导致延迟持续。

2. 不必要的多次文档读写 #

代码中对OneMind文档进行了多次独立的读写操作：先更新processing: true，再读取文档，最后又执行重置更新。每一次操作都需要获取文档锁，增加了锁竞争的频率，进一步加剧延迟。

3. 潜在的索引性能瓶颈（次要） #

对proposedMessages的查询使用了两个where和两个orderBy，如果未提前创建对应的复合索引，Firestore会在运行时动态生成索引（首次查询会触发索引创建），未就绪的索引会导致查询延迟。不过从日志看延迟主要集中在文档更新，所以这是次要因素，但高负载下会叠加影响。

优化建议 #

• 减少单文档并发操作：将activeUsers这类高频更新的字段拆分到子集合，或使用分布式计数器替代数组更新，避免多个客户端同时竞争同一文档的锁。
• 合并文档操作：将对同一文档的多次更新合并为单次update操作，比如在读取文档后一次性设置所有需要更新的字段，减少锁竞争次数。
• 使用事务确保原子性：对于需要原子性的操作（比如读取文档后更新），使用Firestore事务，避免部分更新导致的冲突和重复操作。
• 预创建复合索引：为proposedMessages的查询创建复合索引，避免动态索引带来的延迟。
• 提升GCF资源配置：适当增加GCF的内存分配（比如从256MB提升到512MB），提升函数的处理能力，减少因资源不足导致的延迟。

内容的提问来源于stack exchange，提问作者Joel Castro