M1 Pro环境PySpark读取BigQuery报Connection reset解决方案

开发者阿泽

2026-9-1

问题描述

我尝试通过如下方式读取GBQ表：

from pyspark.sql.functions import col, current_timestamp

def analyze(spark, config, query = None):
    df = spark.read \
        .format("bigquery") \
        .load("projectName.dataset.tablename")
    resultDf = df \
            .filter(col('colName')=='123') \
            .withColumn('processedTS',current_timestamp())
    resultDf.write.mode('overwrite').option("header",True).csv("output/claims")

使用的spark-submit提交命令如下：

spark-submit --name "ApplicationName" --master "local[2]" --deploy-mode "client" --jars jars/spark-bigquery-with-dependencies_2.12-0.25.0.jar,jars/netty-tcnative-2.0.52.Final.jar --verbose main.py --jobName jobnameToRun

已配置GOOGLE_APPLICATION_CREDENTIALS环境变量，指向JSON密钥文件所在路径：

GOOGLE_APPLICATION_CREDENTIALS=/PATH_TO_KEY/file.json

提交应用后出现如下报错：

py4j.protocol.Py4JJavaError: An error occurred while calling o26.load.
    : com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.BigQueryException: Connection reset
            at com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
            at com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.getTable(HttpBigQueryRpc.java:299)
            at com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.BigQueryImpl$18.call(BigQueryImpl.java:778)
            at com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.BigQueryImpl$18.call(BigQueryImpl.java:775)
            at com.google.cloud.spark.bigquery.repackaged.com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
            at com.google.cloud.spark.bigquery.repackaged.com.google.cloud.RetryHelper.run(RetryHelper.java:76)
            at com.google.cloud.spark.bigquery.repackaged.com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
            at com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.BigQueryImpl.getTable(BigQueryImpl.java:774)
            at com.google.cloud.bigquery.connector.common.BigQueryClient.getTable(BigQueryClient.java:119)
            at com.google.cloud.bigquery.connector.common.BigQueryClient.getReadTable(BigQueryClient.java:240)
            at com.google.cloud.spark.bigquery.BigQueryRelationProvider.createRelationInternal(BigQueryRelationProvider.scala:76)
            at com.google.cloud.spark.bigquery.BigQueryRelationProvider.createRelation(BigQueryRelationProvider.scala:45)
            at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:350)
            at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:274)
            at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:245)
            at scala.Option.getOrElse(Option.scala:189)
            at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:245)
            at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:188)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:498)
            at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
            at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
            at py4j.Gateway.invoke(Gateway.java:282)
            at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
            at py4j.commands.CallCommand.execute(CallCommand.java:79)
            at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
            at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
            at java.lang.Thread.run(Thread.java:748)
    Caused by: java.net.SocketException: Connection reset
            at java.net.SocketInputStream.read(SocketInputStream.java:210)
            at java.net.SocketInputStream.read(SocketInputStream.java:141)
            at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
            at sun.security.ssl.InputRecord.read(InputRecord.java:503)
            at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
            at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
            at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
            at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
            at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
            at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
            at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:162)
            at com.google.cloud.spark.bigquery.repackaged.com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:151)
            at com.google.cloud.spark.bigquery.repackaged.com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84)
            at com.google.cloud.spark.bigquery.repackaged.com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012)
            at com.google.cloud.spark.bigquery.repackaged.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514)
            at com.google.cloud.spark.bigquery.repackaged.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
            at com.google.cloud.spark.bigquery.repackaged.com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
            at com.google.cloud.spark.bigquery.repackaged.com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.getTable(HttpBigQueryRpc.java:297)

设备搭载Apple M1 Pro芯片，求解决方案。

解决方案

这是M1系列芯片Mac上常见的架构兼容+SSL握手问题，按以下步骤排查修复：

确认安装的JDK是aarch64原生版本，不要用x86版本通过Rosetta转译运行。执行java -version查看版本信息，如果输出里出现x86_64标识，卸载当前JDK，安装ARM架构适配的JDK 8/11版本（Spark 3.x优先选JDK 8/11，不要用过高版本）。
升级BigQuery连接器版本，当前使用的0.25.0版本对M1芯片的兼容有缺陷，替换为0.29.0及以上版本的spark-bigquery-with-dependencies_2.12包即可，新版本已经修复了Apple Silicon架构下的SSL连接重置问题。不需要额外引入netty-tcnative包，高版本连接器已经内置了适配M1的native依赖，手动添加低版本netty-tcnative反而会造成依赖冲突。
如果本地开启了网络代理，在spark-submit命令里加上JVM代理参数，同时把谷歌云相关域名加入不走代理的列表：

--conf spark.driver.extraJavaOptions="-Dhttp.proxyHost=127.0.0.1 -Dhttp.proxyPort=你的代理端口 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=你的代理端口 -Dhttp.nonProxyHosts=localhost|127.0.0.1|*.googleapis.com|*.google.com"

校验密钥权限：执行gcloud auth activate-service-account --key-file=/PATH_TO_KEY/file.json手动激活服务账号，确认密钥本身有效、对应账号有目标BigQuery表的读取权限。
修复代码语法错误：原代码中withColumn行末尾缺少换行，直接拼接了write语句，正常运行时会触发语法错误，按照问题描述里修正后的代码格式调整即可。

内容的提问来源于stack exchange，提问作者saravana ir