GRPC核心Stream removed错误：场景、成因及技术问题咨询

2023年6月8日更新：问题已解决，根源为防火墙软件的威胁防护功能。最初未在防火墙上发现流量丢弃，启用调试日志后才可见该现象。

问题触发场景 #

• gRPC服务器流场景下，服务器返回客户端数据前耗时超过60秒
• 使用Grpc.Core.Channel（Grpc.Net.Client.GrpcChannel不会触发此错误）
• 在Azure中通过公网IP调用服务器不会触发此问题
• 在Azure中通过内网中心的私网IP调用服务器会触发此问题

咨询问题 #

1. 该问题可能的成因是什么？
2. 为何Grpc.Net.Client.GrpcChannel不会生成此错误，而Grpc.Core.Channel会？
3. 网络中心的哪些配置可能导致该问题？
4. 为何启用日志后连接有时不会被重置？

异常信息 #

Status(StatusCode="Unknown", Detail="Stream removed", DebugException="Grpc.Core.Internal.CoreErrorDetailException: {"created":"@1668425723.993000000","description":"Error received from peer ipv4::5000","file":"......\src\core\lib\surface\call.cc","file_line":953,"grpc_message":"Stream removed","grpc_status":2}")

Wireshark捕获 #

Proto文件 #

syntax = "proto3";

option csharp_namespace = "GrpcDemoServer";

package greet;

service Greeter {
  rpc SayHelloStream (HelloRequest) returns (stream StreamHelloReply);
}

message HelloRequest {  
  string name = 1;
  int32 sleepTime =2;
}

message StreamHelloReply {
    string message = 1;
}

服务器代码 #

服务实现

public override async Task SayHelloStream(HelloRequest request, IServerStreamWriter<StreamHelloReply> responseStream, ServerCallContext context)
{
    for (int i = 1; i <= 10; i++)
    {
        Thread.Sleep(request.SleepTime);//70 seconds
        await responseStream.WriteAsync(new StreamHelloReply
        {
            Message = $"Hello {i}"
        });
    }
}

主程序

public class Program
{
    public static void Main(string[] args)
    {
        CreateHostBuilder(args).Build().Run();
    }

    public static IHostBuilder CreateHostBuilder(string[] args) =>
        Host.CreateDefaultBuilder(args)
            .UseWindowsService()
            .ConfigureWebHostDefaults(webBuilder =>
            {
                webBuilder.ConfigureKestrel(options =>
                {
                    // Setup a HTTP/2 endpoint without TLS.
                    options.ListenAnyIP(5000, o => o.Protocols =
                        HttpProtocols.Http2);

                });
                webBuilder.UseStartup<Startup>();
            });
}

Startup配置

public class Startup
{
    public Startup(IConfiguration configuration)
    {
        Configuration = configuration;
    }
    public IConfiguration Configuration { get; }
   
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddGrpc();
        services.AddApplicationInsightsTelemetry(Configuration["APPINSIGHTS_CONNECTIONSTRING"]);
    }
   
    public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
    {
        app.UseRouting();
                    
        app.UseEndpoints(endpoints =>
        { 
            endpoints.MapGrpcService<GreeterService>();
            endpoints.MapGet("/", async context =>
            {
                await context.Response.WriteAsync("Communication with gRPC endpoints must be made through a gRPC client. To learn how to create a client, visit: https://go.microsoft.com/fwlink/?linkid=2086909");
            });
        });
    }
}

客户端代码 #

var channel = new Channel("<IP ADDRESS>", 1234, ChannelCredentials.Insecure, new[]
{
    new ChannelOption( "grpc.keepalive_time_ms", 1000 ),
    new ChannelOption( "grpc.http2.max_pings_without_data", 0 ),
    new ChannelOption( "grpc.keepalive_permit_without_calls", 1 )
});

var client = new Greeter.GreeterClient(channel);
using var transfer = client.SayHelloStream(GetRequest("Test", 70));
while (await transfer.ResponseStream.MoveNext(CancellationToken.None).ConfigureAwait(false))
{
    var response = transfer.ResponseStream.Current;
    client.SayHello(GetRequest(response.Message, 1));
    Console.WriteLine(response.Message);
}

尝试过的无效配置 #

尝试设置以下ChannelOptions未解决问题：

• grpc.keepalive_time_ms => 1000
• grpc.http2.max_pings_without_data => 0
• grpc.keepalive_permit_without_calls => 1

2023年5月8日更新
最初认为超时时间为60秒，但实际情况更复杂（近似30秒）：

此外，以下特定场景下流有时能正常工作：

• 服务器以gRPC核心Server形式在控制台窗口运行，设置GRPC_TRACE为TCP、GRPC_VERBOSITY为DEBUG
• 客户端在控制台窗口运行，设置GRPC_TRACE为TCP、GRPC_VERBOSITY为DEBUG
• 一旦有一条消息在约30秒的临界点后发送成功，后续所有消息均可成功发送

这似乎表明，TCP消息处理期间为客户端/服务器预留更多时间会影响是否触发错误。

内容的提问来源于stack exchange，提问作者Pascal1986