Firebase云函数读取5k文档超时404错误，如何修复？

问题描述 #

我尝试运行Firebase Cloud Function，该函数需读取5000条Firestore文档，在Insomnia中运行耗时超1分钟，最终崩溃并显示404超时错误。我来自巴西，函数默认区域为美国，但认为更换区域无法解决问题。已尝试优化代码结构，但问题仍未解决。以下是该云函数的代码：

exports.getTodosPedidos = functions.https.onRequest(
   async (req: RequestType, res: ResponseType) => {
    try {
      const idProjeto = req.query.idProjeto;

      if (!idProjeto) {
        onFailureResponseStructure(res, "ID do Projeto não fornecido.", 404);
      }

      const projeto =
        idProjeto &&
        (await admin.firestore().collection("projetos").doc(idProjeto.toString()).get());

      if (projeto != "" && !projeto?.createTime) {
        onFailureResponseStructure(res, "O projeto fornecido não existe.", 405);
      }

      const snapshot = await admin.firestore().collection("pedidos").get();
      const result = snapshot.docs.map(doc => ({ ...doc.data(), id: doc.id }));

      let idsDosPedidos = result.map(elemento => elemento.id);

      let somaDosArquivos = idsDosPedidos.map(async pedidoId => {
        let bucketRef = await admin
          .storage()
          .bucket()
          .getFiles({
            prefix: `${idProjeto}/arquivos/${pedidoId}/`,
          });
        return bucketRef[0].map(elemento => parseInt(elemento.metadata.size));
      });
      const resultado = await Promise.all(somaDosArquivos);

      onSuccessResponseStructure(res, { resultado, idsDosPedidos });
    } catch (error: any) {
      onFailureResponseStructure(res, error.message, 500);
    }
  }
);

const onSuccessResponseStructure = (res: ResponseType, data: any) => {
  return res.status(200).json({
    status: "OK",
    data,
    messages: [],
    statusCode: "200",
  });
};

const onFailureResponseStructure = (
  res: ResponseType,
  messages: string,
  statusCode: number
) => {
  return res.status(statusCode).json({ status: "ERRO", data: [], messages, statusCode });
};

问题根源分析 #

• 无限制并发Storage请求：对5000个订单ID同时发起getFiles请求，触发Firebase限流机制，且耗尽函数资源，导致响应延迟甚至崩溃。
• 一次性拉取大量Firestore数据：直接调用collection("pedidos").get()拉取5000条文档，数据量过大增加网络传输时间和内存占用。
• 错误处理逻辑缺失终止步骤：返回错误响应后未终止函数，后续代码继续运行可能引发额外异常。
• 冗余数据处理：先完整提取所有订单数据再提取ID，浪费内存资源。

优化方案 #

• 控制Storage请求并发数：分批处理请求，避免同时发起大量请求触发限流，降低资源负载。
• Firestore分页读取：分批拉取文档，减少单次请求的数据量，优化内存占用和传输效率。
• 修复错误处理逻辑：返回错误响应后立即终止函数执行，避免无效代码运行。
• 简化数据提取：直接从Firestore快照中提取订单ID，无需先映射完整文档数据。

优化后的代码 #

exports.getTodosPedidos = functions.https.onRequest(
  async (req: RequestType, res: ResponseType) => {
    try {
      const idProjeto = req.query.idProjeto;

      if (!idProjeto) {
        return onFailureResponseStructure(res, "ID do Projeto não fornecido.", 404);
      }

      const projetoDoc = await admin.firestore().collection("projetos").doc(idProjeto.toString()).get();
      
      if (!projetoDoc.exists) {
        return onFailureResponseStructure(res, "O projeto fornecido não existe.", 405);
      }

      // 分页读取Firestore订单ID
      const batchSize = 500;
      let lastVisible = null;
      const idsDosPedidos: string[] = [];

      do {
        let query = admin.firestore().collection("pedidos").limit(batchSize);
        if (lastVisible) {
          query = query.startAfter(lastVisible);
        }
        const snapshot = await query.get();
        snapshot.docs.forEach(doc => idsDosPedidos.push(doc.id));
        lastVisible = snapshot.docs[snapshot.docs.length - 1];
      } while (lastVisible);

      // 分批处理Storage请求，控制并发数
      const concurrencyLimit = 20;
      const resultado: number[][] = [];

      for (let i = 0; i < idsDosPedidos.length; i += concurrencyLimit) {
        const batch = idsDosPedidos.slice(i, i + concurrencyLimit);
        const batchResults = await Promise.all(
          batch.map(async pedidoId => {
            const [files] = await admin.storage().bucket().getFiles({
              prefix: `${idProjeto}/arquivos/${pedidoId}/`,
            });
            return files.map(file => parseInt(file.metadata.size));
          })
        );
        resultado.push(...batchResults);
      }

      onSuccessResponseStructure(res, { resultado, idsDosPedidos });
    } catch (error: any) {
      onFailureResponseStructure(res, error.message, 500);
    }
  }
);

const onSuccessResponseStructure = (res: ResponseType, data: any) => {
  return res.status(200).json({
    status: "OK",
    data,
    messages: [],
    statusCode: "200",
  });
};

const onFailureResponseStructure = (
  res: ResponseType,
  messages: string,
  statusCode: number
) => {
  return res.status(statusCode).json({ status: "ERRO", data: [], messages, statusCode });
};

内容的提问来源于stack exchange，提问作者Rômulo Valentim