使用@google-cloud/pubsub时频繁出现DEADLINE_EXCEEDED错误求助

我们使用@google-cloud/pubsub包监听Google Pub/Sub订阅消息，功能运行正常，但DataDog控制台频繁弹出DEADLINE_EXCEEDED错误，错误栈如下：

Error: 4 DEADLINE_EXCEEDED: Deadline exceeded after 900.000s,LB pick: 0.001s,remote_addr=172.253.122.95:443
at callErrorFromStatus (/app/backend/node_modules/@grpc/grpc-js/src/call.ts:82:17)
at Object.onReceiveStatus (/app/backend/node_modules/@grpc/grpc-js/src/client.ts:706:51)
at Object.onReceiveStatus (/app/backend/node_modules/@grpc/grpc-js/src/client-interceptors.ts:419:48)
at /app/backend/node_modules/@grpc/grpc-js/src/resolving-call.ts:163:24
at processTicksAndRejections (node:internal/process/task_queues:77:11)
for call at
at ServiceClientImpl.makeBidiStreamRequest (/app/backend/node_modules/@grpc/grpc-js/src/client.ts:690:42)
at ServiceClientImpl. (/app/backend/node_modules/@grpc/grpc-js/src/make-client.ts:189:15)
at /dd_tracer/node/dd-trace/packages/datadog-instrumentations/src/grpc/client.js:168:27
at run (/dd_tracer/node/dc-polyfill/patch-channel-store-methods.js:39:16)
at AsyncResourceStorage.run (/dd_tracer/node/dd-trace/packages/datadog-core/src/storage/async_resource.js:69:14)
at /dd_tracer/node/dc-polyfill/patch-channel-store-methods.js:70:18
at Channel.ch.runStores (/dd_tracer/node/dc-polyfill/patch-channel-store-methods.js:48:14)
at callMethod (/dd_tracer/node/dd-trace/packages/datadog-instrumentations/src/grpc/client.js:158:23)
at ServiceClientImpl.wrapped [as streamingPull] (/dd_tracer/node/datadog-instrumentations/src/grpc/client.js:93:12)
at MessageStream._fillOne (/app/backend/node_modules/@google-cloud/pubsub/src/message-stream.ts:332:39)

当前使用的监听代码：

const pubsubClient = new PubSub();

const subscription = pubsubClient.subscription(
  '<subscription-name>',
  {
     flowControl: {
        maxMessages: 100,
     },
  },
);

subscription.on('message', async (msg) => {
    try {
        // Process the message...

        msg.ack();
    } catch (error) {
        console.error('Error in pub/sub', error);
        msg.nack();
    }
});

subscription.on('error', (err) =>
    console.error('Error in pub/sub', err),
);

已知相关社区讨论无明确解决方案，功能正常但希望消除该错误，以下是可行的解决思路：

可能的解决方案 #

1. 调整Pub/Sub客户端的gRPC超时参数 #

错误是streamingPull调用的900秒超时触发的，而Pub/Sub长连接默认需要保持更久。初始化客户端时手动覆盖gRPC配置：

const pubsubClient = new PubSub({
  grpc: {
    options: {
      'grpc.keepalive_time_ms': 30000, // 每30秒发送保活包
      'grpc.keepalive_timeout_ms': 5000, // 保活超时5秒
      'grpc.keepalive_permit_without_calls': 1, // 无调用时允许保活
      'grpc.default_deadline_ms': 3600000 // 把默认超时调整为3600秒
    }
  }
});

2. 排除DataDog对Pub/Sub gRPC调用的追踪 #

从错误栈看，Datadog的gRPC instrumentation干扰了长连接超时处理。可以在tracer初始化时排除指定方法：

require('dd-trace').init({
  instrumentations: {
    grpc: {
      client: {
        filter: (service, method) => {
          return !(service === 'google.pubsub.v1.Subscriber' && method === 'StreamingPull');
        }
      }
    }
  }
});

或者通过环境变量配置：

export DD_GRPC_CLIENT_EXCLUDE_METHODS="google.pubsub.v1.Subscriber/StreamingPull"

3. 升级依赖版本 #

确保@google-cloud/pubsub、@grpc/grpc-js和dd-trace为最新稳定版，新版本可能修复了兼容性问题：

npm update @google-cloud/pubsub @grpc/grpc-js dd-trace

4. 优化消息处理逻辑 #

避免消息处理回调阻塞事件循环，把耗时任务拆分到异步队列，确保ack/nack尽快执行：

• 拆分耗时处理逻辑到独立的worker进程或任务队列
• 确保错误处理覆盖所有分支，避免未处理Promise阻塞

5. 过滤特定超时错误 #

如果以上方法无效，可在错误监听中过滤该特定错误，避免DataDog频繁告警：

subscription.on('error', (err) => {
  if (err.code === 4 && err.details.includes('Deadline exceeded after 900.000s')) {
    return;
  }
  console.error('Error in pub/sub', err);
});

内容的提问来源于stack exchange，提问作者Snehanshu Phukon