Pub/Sub推送订阅未确认消息、消息重复投递(Cloud Run部署)
基于Cloud Run部署的Pub/Sub推送订阅ACK率低、消息重复投递问题
部署配置
- 订阅A为*推送(push)*类型,会以POST请求形式将消息发送到Cloud Run托管的服务端点
- 服务业务流程:接收订阅A推送的消息→完成消息处理→将处理结果发布到Topic B→向订阅A返回200状态码完成确认
- 单条消息全流程处理耗时约1.5秒,理论上订阅A每处理1条消息,Topic B应新增1条对应消息
- 订阅A的确认截止时间(Acknowledgement deadline)设置为600秒
- Cloud Run服务的请求超时时间设置为300秒
实现代码
服务启动代码
const express = require('express'); const bodyParser = require('body-parser'); const _ = require('lodash'); const startBrowser = require('./startBrowser'); const tab = require('./tab'); const createMessage = require('./publishMessage'); const domain = 'https://example.com'; require('dotenv').config(); const app = express(); app.use(bodyParser.json()); const port = process.env.PORT || 8080; app.listen(port, async () => { console.log('Listening on port', port); });
核心请求处理代码
app.post('/', async (req, res) => { // Define the success and fail functions, that respond status 200 and 500 respectively const failed = () => res.status(500).send(); const completed = async () => { const response = await res.status(200).send(); if (response && res.writableEnded) { console.log('successfully responded 200'); } }; //Process the data coming from Subscription A let pubsubMessage = decodeBase64Json(req.body.message.data); let parsed = await processor(pubsubMessage); //Post the processed data to topic B let messageId = await postParsedData(parsed); if (messageId) { // ACK the message once the data has been processed and posted to topic B. completed(); } else { console.log('Didnt get a message id'); // failed(); } }); //define the functions that post data to Topic B const postParsedData = async (parsed) => { if (!_.isEmpty(parsed)) { const topicName = 'topic-B'; const messageIdInternal = await createMessage(parsed, topicName); }; return messageId; } else { console.log('Parsed is Empty'); return null; } }; function decodeBase64Json(data) { return JSON.parse(Buffer.from(data, 'base64').toString()); }
异常现象
- 单实例Cloud Run处理单条消息耗时约1.5秒,日志中每1.5秒可观测到一次200响应成功记录,折算单实例每小时可处理约2400条消息
- Topic B每小时实际新增约2400条消息,和理论处理量匹配,但订阅A的确认率仅约每小时200条
- 大量消息因未在确认截止时间内收到有效ACK,被Pub/Sub重复投递
- 已尝试将返回200的时机前置到消息发布到Topic B前、甚至消息解析前,问题仍未解决
- 已通过待处理、已处理消息量的对比记录验证:已处理消息量远大于已确认的待处理消息量,二者不符合理论上1:1的对应关系
排查与解决方案
- 向GCP官方支持提交问题,官方环境下无法直接复现该异常
- 经多轮排查确认,当Cloud Run工作实例数量充足时,问题完全消失
- 最终解决方案:扩容增加Cloud Run工作实例数量
内容的提问来源于stack exchange,提问作者Felipe Lujan
相关产品推荐
相关产品推荐

