Dataproc作业JDBC写入Cloud Spanner遇ProviderNotFoundException错误

问题详情 #

• 同一Dataproc作业中，使用Spanner Connector读取Cloud Spanner表正常，排除权限问题；但因Connector不支持写入，改用JDBC写入时报错
• Compute Engine默认服务账号已拥有角色：BigQuery Data Owner、BigQuery Job User、Cloud Spanner Database Admin、Compute OS Login、Dataproc Editor、Dataproc Worker、Service Account User、Storage Admin
• 使用环境：Dataproc Serverless 2.1，尝试过Spanner JDBC Jar 2.14.2、2.4及其他版本，不同版本报错不同
• 已尝试的解决方式：构建FatJar、将JDBC Jar存放在GCS桶作为运行时依赖通过gcloud命令添加，均无效

报错信息 #

Caused by: com.google.cloud.spanner.SpannerException: UNKNOWN: io.grpc.netty.NettyChannelProvider: Unsupported credential type: io.grpc.netty.shaded.io.grpc.netty.NettyChannelCredentials; io.grpc.netty.UdsNettyChannelProvider: does not support 1 or more of [class java.net.InetSocketAddress]
        at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:291)
        at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:297)
        at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:61)
        at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:181)
        at com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:110)
        at com.google.cloud.spanner.spi.v1.GapicSpannerRpc.<init>(GapicSpannerRpc.java:484)
        at com.google.cloud.spanner.spi.v1.GapicSpannerRpc.<init>(GapicSpannerRpc.java:271)
        at com.google.cloud.spanner.SpannerOptions$DefaultSpannerRpcFactory.create(SpannerOptions.java:475)
        at com.google.cloud.spanner.SpannerOptions$DefaultSpannerRpcFactory.create(SpannerOptions.java:470)
        at com.google.cloud.ServiceOptions.getRpc(ServiceOptions.java:602)
        at com.google.cloud.spanner.SpannerOptions.getSpannerRpcV1(SpannerOptions.java:1417)
        at com.google.cloud.spanner.SpannerImpl.<init>(SpannerImpl.java:133)
        at com.google.cloud.spanner.SpannerOptions$DefaultSpannerFactory.create(SpannerOptions.java:465)
        at com.google.cloud.spanner.SpannerOptions$DefaultSpannerFactory.create(SpannerOptions.java:460)
        at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:582)
        at com.google.cloud.spanner.connection.SpannerPool.createSpanner(SpannerPool.java:361)
        at com.google.cloud.spanner.connection.SpannerPool.getSpanner(SpannerPool.java:302)
        at com.google.cloud.spanner.connection.ConnectionImpl.<init>(ConnectionImpl.java:254)
        at com.google.cloud.spanner.connection.ConnectionOptions.getConnection(ConnectionOptions.java:1141)
        at com.google.cloud.spanner.jdbc.AbstractJdbcConnection.<init>(AbstractJdbcConnection.java:63)
        at com.google.cloud.spanner.jdbc.JdbcConnection.<init>(JdbcConnection.java:67)
        at com.google.cloud.spanner.jdbc.JdbcDriver.connect(JdbcDriver.java:196)
        ... 48 more
Caused by: io.grpc.ManagedChannelRegistry$ProviderNotFoundException: io.grpc.netty.NettyChannelProvider: Unsupported credential type: io.grpc.netty.shaded.io.grpc.netty.NettyChannelCredentials; io.grpc.netty.UdsNettyChannelProvider: does not support 1 or more of [class java.net.InetSocketAddress]
        at io.grpc.ManagedChannelRegistry.newChannelBuilder(ManagedChannelRegistry.java:206)
        at io.grpc.ManagedChannelRegistry.newChannelBuilder(ManagedChannelRegistry.java:157)
        at io.grpc.Grpc.newChannelBuilder(Grpc.java:101)
        at io.grpc.Grpc.newChannelBuilderForAddress(Grpc.java:111)
        at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:387)
        at com.google.api.gax.grpc.ChannelPool.<init>(ChannelPool.java:107)
        at com.google.api.gax.grpc.ChannelPool.create(ChannelPool.java:85)
        at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:243)
        at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:237)
        at com.google.api.gax.rpc.ClientContext.create(ClientContext.java:226)
        at com.google.cloud.spanner.v1.stub.GrpcSpannerStub.create(GrpcSpannerStub.java:243)
        at com.google.cloud.spanner.spi.v1.GapicSpannerRpc.<init>(GapicSpannerRpc.java:370)

相关代码 #

JDBC写入（异常代码） #

String spannerConnectionUrl = String.format(
                "jdbc:cloudspanner:/projects/%s/instances/%s/databases/%s?lenient=true",
                spannerProjectId, spannerInstanceId, spannerDatabase
        );

log.info("Spanner connection url {}",spannerConnectionUrl);
testDF.write()
        .option("dbtable", "test_activity")
        .format("jdbc")
        .option("driver", "com.google.cloud.spanner.jdbc.JdbcDriver")
        .option("url", spannerConnectionUrl)
        .mode("overwrite")
        .save();

Spanner Connector读取（正常代码） #

// Read from Cloud Spanner using connector
Dataset<Row> testDF = spark.read()
        .format("cloud-spanner")
        .option("table", "test_activity")
        .option("instanceId", spannerInstanceId)
        .option("databaseId", spannerDatabase)
        .option("projectId", spannerProjectId)
        .load()
        .filter(col("id").equalTo(id));

内容的提问来源于stack exchange，提问作者Ani