GCP Cloud Tasks创建时间歇性触发Grpc.Core.RpcException求助

问题背景 #

在GCP上创建Cloud Task时，间歇性触发Grpc.Core.RpcException异常，每日数十万请求中约出现100次。当前环境为.NET 8，使用Google.Cloud.Tasks.V2 3.5.0 NuGet包，需彻底解决该问题而非仅依赖重试逻辑。

异常详情 #

Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error starting gRPC call. HttpRequestException: An error occurred while sending the request. IOException: An HTTP/2 connection could not be established because the server did not complete the HTTP/2 handshake. ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Net.Security.SslStream'.", DebugException="System.Net.Http.HttpRequestException: An error occurred while sending the request.")
 ---> System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: An HTTP/2 connection could not be established because the server did not complete the HTTP/2 handshake.
 ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Net.Security.SslStream'.
   at System.Net.Security.SslStream.<ThrowIfExceptional>g__ThrowExceptional|126_0(ExceptionDispatchInfo e)
   at System.Net.Http.Http2Connection.SetupAsync(CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.Http2Connection.SetupAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConstructHttp2ConnectionAsync(Stream stream, HttpRequestMessage request, IPEndPoint remoteEndPoint, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnectionPool.ConstructHttp2ConnectionAsync(Stream stream, HttpRequestMessage request, IPEndPoint remoteEndPoint, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp2ConnectionAsync(QueueItem queueItem)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at Grpc.Net.Client.Balancer.Internal.BalancerHttpHandler.SendAsync(HttpRequestMessage request, Boolean cancellationToken)
   at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)
   --- End of inner exception stack trace ---
   at Google.Api.Gax.Grpc.ApiCallRetryExtensions.<>c__DisplayClass0_0`2.<<WithRetry>b__0>d.MoveNext()
--- End of stack trace from previous location ---

触发异常的代码 #

public async System.Threading.Tasks.Task Publish(CloudTaskIntegrationEvent taskIntegrationEvent)
{
    var parent = new QueueName(_projectId, _location, taskIntegrationEvent.Queue);
    var baseUrl = taskIntegrationEvent.BaseUrl ?? _baseUrl;

    var fullEndpointUrl = $"{baseUrl}/{taskIntegrationEvent.Endpoint}";

    _logger.LogInformation("Full Endpoint Url for Cloud Task: {FullEndPointUrl}", fullEndpointUrl);

    var httpRequest = new HttpRequest
    {
        HttpMethod = HttpMethod.Post,
        Url = fullEndpointUrl,
        Body = ByteString.CopyFromUtf8(taskIntegrationEvent.Payload),
        Headers =
        {
            {"Content-Type", "application/json"}
        }
    };

    if(_gcpServiceAccountId != null)
    {
        var oidcToken = new OidcToken
        {
            ServiceAccountEmail = _gcpServiceAccountId
        };
        httpRequest.OidcToken = oidcToken;
    }

    var task = new Google.Cloud.Tasks.V2.Task
    {
        HttpRequest = httpRequest
    };

    if (!string.IsNullOrWhiteSpace(taskIntegrationEvent.TaskName))
        task.Name = $"{parent}/tasks/{taskIntegrationEvent.TaskName}";

    try
    {
        _logger.LogInformation("Creating cloud task in {Queue} with payload {Payload}", taskIntegrationEvent.Queue, taskIntegrationEvent.Payload);

        var request = new CreateTaskRequest
        {
            Parent = parent.ToString(),
            Task = task
        };

        request.Task.ScheduleTime = Timestamp.FromDateTime(
            DateTime.UtcNow.AddSeconds(taskIntegrationEvent.Delay + _defaultDelay));
        var response = await _cloudTasksClient.CreateTaskAsync(request);
        _logger.LogInformation("Task {Name} created", response.Name);
    }
    catch (Exception e)
    {
        _logger.LogError(e, "Error while creating could task for task |  {Queue}, {EventType}, {Payload}", taskIntegrationEvent.Queue, taskIntegrationEvent.EventType, taskIntegrationEvent.Payload);
    }
}

解决方案 #

1. 升级GCP SDK及Grpc依赖包 #

Google.Cloud.Tasks.V2 3.5.0版本存在.NET 8环境下的HTTP/2兼容性问题，直接升级至最新稳定版（如4.x系列），同时同步升级Grpc.Net.Client、Google.Api.Gax.Grpc等关联依赖包，新版本通常修复了连接池管理、HTTP/2握手的相关bug。

2. 自定义CloudTasksClient连接池配置 #

默认连接池配置在高并发场景下易出现连接过早释放的问题，手动配置Grpc通道参数，优化连接生命周期：

// 单例模式创建CloudTasksClient，避免重复实例化
private static CloudTasksClient _cloudTasksClient;

static YourTaskPublisher()
{
    var socketHandler = new SocketsHttpHandler
    {
        PooledConnectionIdleTimeout = TimeSpan.FromMinutes(5), // 延长空闲连接回收时间
        KeepAlivePingDelay = TimeSpan.FromSeconds(30), // 定期发送保活包维持连接
        KeepAlivePingTimeout = TimeSpan.FromSeconds(5),
        EnableMultipleHttp2Connections = true // 高并发下允许创建多个HTTP/2连接
    };

    var channelOptions = new GrpcChannelOptions
    {
        HttpHandler = socketHandler,
        ConnectTimeout = TimeSpan.FromSeconds(10),
        MaxRetryAttempts = 0 // 先禁用SDK自带重试，按需调整
    };

    _cloudTasksClient = CloudTasksClient.Create(channelOptions);
}

关键注意点：确保CloudTasksClient以单例方式使用，频繁创建实例会导致连接池混乱，加剧异常。

3. 临时切换至HTTP/1.1排查问题 #

若升级后异常仍存在，可强制客户端使用HTTP/1.1验证是否为HTTP/2兼容性问题：

var channelOptions = new GrpcChannelOptions
{
    HttpHandler = new SocketsHttpHandler
    {
        AllowHttp2 = false
    }
};
_cloudTasksClient = CloudTasksClient.Create(channelOptions);

此为临时排查方案，长期建议修复HTTP/2问题以保障高并发性能。

4. 调整.NET 8 HTTP/2系统配置 #

通过环境变量调整.NET 8的HTTP/2默认行为：

• 设置DOTNET_SYSTEM_NET_HTTP_SOCKETSHTTPHANDLER_POOLEDCONNECTIONIDLETIMEOUT为300（单位：秒），延长连接池空闲连接的存活时间
• 设置DOTNET_SYSTEM_NET_HTTP_SOCKETSHTTPHANDLER_KEEPALIVEPINGDELAY为30（单位：秒），定期发送保活包

5. 排查GCP端限制 #

• 检查Cloud Task的API配额，确保未达到速率限制或连接数上限
• 若使用VPC网络访问，确认防火墙规则允许443端口的HTTPS流量，且支持ALPN协议（HTTP/2依赖）

总结 #

优先通过升级依赖包解决底层兼容性问题，其次优化客户端连接池配置，最后排查环境与GCP端的限制，这些措施可从根源上消除此类间歇性连接异常。

内容的提问来源于stack exchange，提问作者Dovid Rabinowitz