Databricks使用spark.read连接SQL Server报KrbException无法定位默认领域错误如何解决

我尝试在Databricks notebook中连接SQL数据库，代码如下：

jdbcDF = spark.read \
    .format("com.microsoft.sqlserver.jdbc.spark") \
    .option("url", "jdbc:sqlserver://dbs-sdp-sql-dev.database.windows.net:1433;database=dm_test;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;Authentication=ActiveDirectoryIntegrated") \
    .option("dbtable", "Cars") \
    .option("Driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver").load()

运行后收到如下报错：

IllegalArgumentException: KrbException: Cannot locate default realm

完整错误堆栈如下：

IllegalArgumentException                  Traceback (most recent call last)
<command-1799101041598361> in <module>
----> 1 jdbcDF = spark.read \
      2         .format("com.microsoft.sqlserver.jdbc.spark") \
      3         .option("url", "jdbc:sqlserver://dbs-sdp-sql-dev.database.windows.net:1433;database=dm_test;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;Authentication=ActiveDirectoryIntegrated") \
      4         .option("dbtable", "Cars") \
      5         .option("Driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver").load()

/databricks/spark/python/pyspark/sql/readwriter.py in load(self, path, format, schema, **options)
    208             return self._df(self._jreader.load(self._spark._sc._jvm.PythonUtils.toSeq(path)))
    209         else:
--> 210             return self._df(self._jreader.load())
    211 
    212     def json(self, path, schema=None, primitivesAsString=None, prefersDecimal=None,

/databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py in __call__(self, *args)
   1302 
   1303         answer = self.gateway_client.send_command(command)
-> 1304         return_value = get_return_value(
   1305             answer, self.gateway_client, self.target_id, self.name)
   1306 

/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
    121                 # Hide where the exception came from that shows a non-Pythonic
    122                 # JVM exception message.
--> 123                 raise converted from None
    124             else:
    125                 raise

IllegalArgumentException: KrbException: Cannot locate default realm

我已尝试在集群安装python-krb5，安装失败；也尝试安装requests-kerberos==0.12.0，同样安装失败。

该错误的核心原因是JDBC URL中使用的Authentication=ActiveDirectoryIntegrated认证方式依赖本地Kerberos域配置，而Databricks默认运行环境没有配置对应Azure AD的Kerberos域信息，不需要安装krb5相关依赖，直接替换认证方式即可修复：

• 方案1：Azure AD服务主体认证（生产场景推荐）
  提前创建拥有Azure SQL访问权限的服务主体，修改连接代码即可：

  jdbcDF = spark.read \
      .format("com.microsoft.sqlserver.jdbc.spark") \
      .option("url", "jdbc:sqlserver://dbs-sdp-sql-dev.database.windows.net:1433;database=dm_test;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30") \
      .option("dbtable", "Cars") \
      .option("AADSecurePrincipalId", "服务主体应用ID") \
      .option("AADSecurePrincipalSecret", "服务主体密钥") \
      .option("tenantId", "Azure租户ID") \
      .option("Driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver").load()
  

• 方案2：SQL账号密码认证（测试场景推荐）
  若开启了SQL Server认证模式，直接配置账号密码即可：

  jdbcDF = spark.read \
      .format("com.microsoft.sqlserver.jdbc.spark") \
      .option("url", "jdbc:sqlserver://dbs-sdp-sql-dev.database.windows.net:1433;database=dm_test;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30") \
      .option("dbtable", "Cars") \
      .option("user", "SQL账号") \
      .option("password", "SQL密码") \
      .option("Driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver").load()
  

• 方案3：Databricks托管身份认证
  若集群已配置有权限访问SQL库的托管身份，直接修改认证参数即可：

  jdbcDF = spark.read \
      .format("com.microsoft.sqlserver.jdbc.spark") \
      .option("url", "jdbc:sqlserver://dbs-sdp-sql-dev.database.windows.net:1433;database=dm_test;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;Authentication=ActiveDirectoryMSI") \
      .option("dbtable", "Cars") \
      .option("Driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver").load()
  

内容的提问来源于stack exchange，提问作者Markus Fjellheim