Heroku上使用MassTransit时初始GRPC请求超时问题排查

环境与配置 #

• 两个基于gRPC的MassTransit总线，分别由两个不同的.NET 7 WebApplication托管，运行在同一Docker进程中并部署到Heroku
• 对外暴露REST API端点POST /test-grpc用于发起gRPC请求
• 使用版本：MassTransit/MassTransit.Grpc 8.1.3、.NET 7
• 总线核心配置（两个应用仅端口不同）：

services.AddMassTransit<IGrpcBus>(cfg =>
{
    cfg.UsingGrpc((context, grpcCfg) =>
    {
        grpcCfg.Host(h =>
        {
            h.Host = "127.0.0.1";
            h.Port = 19001;

            h.AddServer(new Uri("http://127.0.0.1:19000"));
        });

        grpcCfg.AutoStart = true;

        grpcCfg.ConfigureEndpoints(context);

    });
});

• REST端点实现代码：

app.MapPost("test-grpc", async () =>
{
    using var scope = app1.Services.CreateScope();
    var bus = scope.ServiceProvider.GetRequiredService<IGrpcBus>();
    var client = bus.CreateRequestClient<GrpcMessage>();
    var response = await client.GetResponse<GrpcResponse>(new GrpcMessage() { Value = "request_123" });
    return response.Message.ResponseValue;
});

问题现象 #

• Heroku启动后日志显示两个总线已相互连接，但向POST <heroku_app_external_address>/test-grpc发起请求时，前2-3次请求必然超时，抛出MassTransit.RequestTimeoutException
• 超时请求后，后续所有gRPC请求均可正常执行，直到Heroku dyno重启
• 无论应用启动后立即请求还是等待5分钟再请求，首次几次请求都会超时
• 本地Docker环境无此问题

相关日志 #

启动连接日志 #

Jan 26 16:37:42  [INF APP_2] gRPC Connect: http://127.0.0.1:19001/ (MassTransit) {"Module":"METRICS","SourceContext":"MassTransit","Level":"Information","Timestamp":"2024-01-26T15:37:42.4294311+00:00"}
Jan 26 16:37:42  [INF APP_1] gRPC Connect: http://127.0.0.1:19000/ (MassTransit) {"Module":"JIRA","SourceContext":"MassTransit","Level":"Information","Timestamp":"2024-01-26T15:37:42.4573197+00:00"}
Jan 26 16:37:43  [INF APP_1] gRPC Connected: http://127.0.0.1:19000/ (MassTransit) {"Module":"JIRA","SourceContext":"MassTransit","Level":"Information","Timestamp":"2024-01-26T15:37:43.1110933+00:00"}
Jan 26 16:37:43  [INF APP_2] gRPC Connected: http://127.0.0.1:19001/ (MassTransit) {"Module":"METRICS","SourceContext":"MassTransit","Level":"Information","Timestamp":"2024-01-26T15:37:43.1185939+00:00"}
Jan 26 16:37:43  [INF APP_2] Bus started: http://127.0.0.1:19000/ (MassTransit) {"Module":"METRICS","SourceContext":"MassTransit","Level":"Information","Timestamp":"2024-01-26T15:37:43.1625409+00:00"}
Jan 26 16:37:43  [INF APP_1] Bus started: http://127.0.0.1:19001/ (MassTransit) {"Module":"JIRA","SourceContext":"MassTransit","Level":"Information","Timestamp":"2024-01-26T15:37:43.1641062+00:00"}

超时错误日志 #

MassTransit.RequestTimeoutException: 
Timeout waiting for response, RequestId: 02000000-e137-5299-37e7-08dc1e865064\n   
at MassTransit.Clients.ResponseHandlerConnectHandle`1.GetTask() in /_/src/MassTransit/Clients/ResponseHandlerConnectHandle.cs:line 58\n   
at MassTransit.Clients.RequestClient`1.GetResponseInternal[T](SendRequestCallback request, CancellationToken cancellationToken, RequestTimeout timeout, RequestPipeConfiguratorCallback`1 callback) in /_/src/MassTransit/Clients/RequestClient.cs:line 195\n   
at Program.<<<Main>$>b__4>d.MoveNext() in /app/src/Gateway/Gateway.Host/Program.cs:line 69\n--- End of stack trace from previous location ---\n   
at Microsoft.AspNetCore.Http.RequestDelegateFactory.<ExecuteTaskOfString>g__ExecuteAwaited|113_0(Task`1 task, HttpContext httpContext)\n   
at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)\n   
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

怀疑方向 #

1. Heroku环境下gRPC初始连接耗时超过默认超时阈值，需要调整connect-timeout，但未找到MassTransit对应配置项
2. 即使设置AutoStart = true，日志显示已连接，但实际gRPC连接可能仍在首次请求时才建立（设置AutoStart = false时日志同样显示连接，不符合预期）
3. IGrpcBus是第二个总线（第一个为RabbitMQ总线），无法通过DI获取其IBusControl实例，无法等待总线健康状态，不确定是否影响连接稳定性

解决方案建议 #

1. 显式设置请求超时时间 #

创建RequestClient时指定更长的超时时间，覆盖默认值：

var client = bus.CreateRequestClient<GrpcMessage>(TimeSpan.FromSeconds(10)); // 根据实际场景调整时长
var response = await client.GetResponse<GrpcResponse>(new GrpcMessage() { Value = "request_123" });

2. 预热gRPC连接 #

在应用启动完成后，主动发起一次轻量请求预热连接，避免首次用户请求超时：

// 放置在Program.cs的app.Run()之前
using var scope = app1.Services.CreateScope();
var bus = scope.ServiceProvider.GetRequiredService<IGrpcBus>();
var client = bus.CreateRequestClient<GrpcMessage>(TimeSpan.FromSeconds(10));
try
{
    await client.GetResponse<GrpcResponse>(new GrpcMessage() { Value = "warmup" });
}
catch (Exception)
{
    // 忽略预热失败，后续用户请求会自动重试
}

3. 配置gRPC底层通道参数 #

通过MassTransit的gRPC配置扩展，直接设置gRPC通道的连接超时：

cfg.UsingGrpc((context, grpcCfg) =>
{
    grpcCfg.Host(h =>
    {
        h.Host = "127.0.0.1";
        h.Port = 19001;
        h.AddServer(new Uri("http://127.0.0.1:19000"));
    });

    // 配置gRPC通道选项
    grpcCfg.ConfigureChannel(options =>
    {
        options.ConnectTimeout = TimeSpan.FromSeconds(5); // 设置连接超时
        options.MaxRetryAttempts = 3; // 增加重试次数
    });

    grpcCfg.AutoStart = true;
    grpcCfg.ConfigureEndpoints(context);
});

4. 手动确认总线启动状态 #

对于第二个总线，可通过IBusFactory获取IBusControl实例并等待启动完成：

// 在服务注册后执行
var busControl = services.BuildServiceProvider().GetRequiredService<IBusFactory<IGrpcBus>>().CreateBus();
await busControl.StartAsync();
// 可将busControl存储到全局变量，在应用关闭时调用StopAsync()

内容的提问来源于stack exchange，提问作者user606521