将Spark DataFrame插入SQL Server时遇ClassNotFoundException错误求助

我使用PySpark将Spark DataFrame写入SQL Server数据库，IDE为PyCharm，代码片段如下：

server_name = "SERVER"
database_name = "DB"
url = "jdbc:sqlserver://{"+server_name+"} " + ";" + "databaseName=" + database_name + ";"

table_name = "Table_Temp"
username = "USER"
password = "PASS" 


df_target.write \
    .format("com.microsoft.sqlserver.jdbc.spark") \
    .mode("overwrite") \
    .option("url", url) \
    .option("dbtable", table_name) \
    .option("user", username) \
    .option("password", password) \
    .save()

执行代码后出现如下错误：

Traceback (most recent call last):
  File "E:\python\SCD2.py", line 129, in <module>
    df_target.write \
  File "E:\spark-3.3.1-bin-hadoop3\python\lib\pyspark.zip\pyspark\sql\readwriter.py", line 966, in save
  File "E:\spark-3.3.1-bin-hadoop3\python\lib\py4j-0.10.9.5-src.zip\py4j\java_gateway.py", line 1321, in __call__
  File "E:\spark-3.3.1-bin-hadoop3\python\lib\pyspark.zip\pyspark\sql\utils.py", line 190, in deco
  File "E:\spark-3.3.1-bin-hadoop3\python\lib\py4j-0.10.9.5-src.zip\py4j\protocol.py", line 326, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o198.save.
: java.lang.ClassNotFoundException: 
Failed to find data source: com.microsoft.sqlserver.jdbc.spark. Please find packages at
https://spark.apache.org/third-party-projects.html
       
    at org.apache.spark.sql.errors.QueryExecutionErrors$.failedToFindDataSourceError(QueryExecutionErrors.scala:587)
    at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:675)
    at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSourceV2(DataSource.scala:725)
    at org.apache.spark.sql.DataFrameWriter.lookupV2Provider(DataFrameWriter.scala:864)
    at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:256)
    at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:247)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
    at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: com.microsoft.sqlserver.jdbc.spark.DefaultSource
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$lookupDataSource$5(DataSource.scala:661)
    at scala.util.Try$.apply(Try.scala:213)
    at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$lookupDataSource$4(DataSource.scala:661)
    at scala.util.Failure.orElse(Try.scala:224)
    at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:661)

我已安装Spark，但不清楚问题所在，恳请帮忙排查解决。

这个错误的核心是Spark缺少Microsoft SQL Server的Spark连接器依赖包，com.microsoft.sqlserver.jdbc.spark是微软提供的Spark与SQL Server交互的数据源，需要额外引入对应jar包才能使用。以下是两种可行的解决方式：

方式1：启动时指定依赖包 #

在启动PySpark或运行脚本时，通过配置参数引入微软的Spark SQL Server连接器包，需确保版本与你的Spark、Scala版本兼容（Spark 3.3.1对应Scala 2.12）。

方式1.1：通过命令行启动PySpark #

pyspark --packages com.microsoft.azure:spark-mssql-connector_2.12:1.2.0

方式1.2：在代码中配置SparkSession #

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("SQLServerWrite") \
    .config("spark.jars.packages", "com.microsoft.azure:spark-mssql-connector_2.12:1.2.0") \
    .getOrCreate()

方式2：手动添加jar包到Spark目录 #

1. 下载对应版本的spark-mssql-connector jar包及依赖的mssql-jdbc jar包；
2. 将下载好的jar包复制到你的Spark安装目录下的jars文件夹（示例路径：E:\spark-3.3.1-bin-hadoop3\jars）；
3. 重启PyCharm及相关Spark服务，让依赖生效。

额外优化建议 #

• 修正JDBC URL的拼接方式，避免多余空格问题：

url = f"jdbc:sqlserver://{server_name};databaseName={database_name};"

• 确认SQL Server版本与连接器版本兼容，避免出现兼容性异常。

内容的提问来源于stack exchange，提问作者user