Spark Scala Azure挂载代码迁移至PySpark实现容器挂载校验

需求：实现检查ADLS容器是否已挂载，若未挂载则执行挂载操作，供团队成员集中使用。

原Scala代码 #

val configs = Map(
  "fs.azure.account.auth.type" -> "OAuth",
  "fs.azure.account.oauth.provider.type" -> "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
  "fs.azure.account.oauth2.client.id" -> dbutils.secrets.get(scope = "sample-scope", key = "sample--client-id"),
  "fs.azure.account.oauth2.client.secret" -> dbutils.secrets.get(scope = "sample-scope", key = "sample-client-secret"),
  "fs.azure.account.oauth2.client.endpoint" -> "https://login.microsoftonline.com/33333df5c2-953a-444444444/oauth2/token"
)

val adlsPath = "abfss://container@storageaccount.dfs.core.windows.net/"
val mountPoint = "/mnt/containername"

if (dbutils.fs.mounts.map(mnt => mnt.mountPoint).contains(mountPoint)) {
  println(mountPoint + " 已挂载")
}
else {
  println(mountPoint + " 未挂载，开始执行挂载")
  try {
    dbutils.fs.mount(
      source = adlsPath,
      mountPoint = mountPoint,
      extraConfigs = configs)
  }
  catch {
    case e: java.rmi.RemoteException => {
      println("挂载 " + adlsPath + " 时遇到异常")
    }
  }
}

转换后的PySpark代码 #

configs = {
    "fs.azure.account.auth.type": "OAuth",
    "fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
    "fs.azure.account.oauth2.client.id": dbutils.secrets.get(scope="sample-scope", key="sample--client-id"),
    "fs.azure.account.oauth2.client.secret": dbutils.secrets.get(scope="sample-scope", key="sample-client-secret"),
    "fs.azure.account.oauth2.client.endpoint": "https://login.microsoftonline.com/33333df5c2-953a-444444444/oauth2/token"
}

adls_path = "abfss://container@storageaccount.dfs.core.windows.net/"
mount_point = "/mnt/containername"

if any(mnt.mountPoint == mount_point for mnt in dbutils.fs.mounts()):
    print(f"{mount_point} 已挂载")
else:
    print(f"{mount_point} 未挂载，开始执行挂载")
    try:
        dbutils.fs.mount(
            source=adls_path,
            mount_point=mount_point,
            extra_configs=configs
        )
    except Exception as e:
        print(f"挂载 {adls_path} 时遇到异常")

关键转换说明 #

• Scala的Map结构转为Python原生字典dict
• 变量命名从驼峰式调整为Python常用的下划线分隔式
• 挂载检查逻辑从map+contains改为Python的any()生成器表达式，执行效率更高
• 异常捕获从Scala模式匹配改为Python的except语法，若需严格匹配原代码的特定异常，可导入java.rmi.RemoteException后指定捕获
• 字符串输出采用Python的f-string语法，更简洁直观

内容的提问来源于stack exchange，提问作者Developer Rajinikanth