Databricks调用dbutils.secrets.get获取S3密钥报NoSuchElementException如何解决

云上老周

2026-9-30

问题原因

报错核心是dbutils.secrets.get仅允许在Databricks Driver节点执行，你把调用逻辑写在了foreachPartition内部，而foreachPartition内的代码会分发到Executor节点运行，Executor节点不存在dbutils的上下文实例，因此调用时抛出空值获取异常。你直接传明文可以运行，是因为绕开了Executor端调用dbutils的逻辑。

解决方法

把密钥的获取逻辑移到Driver端执行，再将拿到的密钥传入foreachPartition内部使用即可，修正后的代码如下：

// Driver端先获取密钥，不会分发到Executor
val AccessKey = dbutils.secrets.get(scope = "ADB_Scope", key = "AccessKey-ID")
val SecretKey = dbutils.secrets.get(scope = "ADB_Scope", key = "AccessKey-Secret") 

my_dataframe.rdd.foreachPartition(partition => {
        // 把拿到的密钥带到Executor里初始化S3客户端，不会在代码和日志中明文留存
        val creds = new BasicAWSCredentials(AccessKey, SecretKey)
        val clientRegion: Regions = Regions.US_EAST_1
        val s3client  = AmazonS3ClientBuilder.standard()
        .withRegion(clientRegion)
        .withCredentials(new AWSStaticCredentialsProvider(creds))
        .build()
          partition.foreach(x => {
            val objectKey = x.getString(0)
            val i = s3client.getObject(s3bucketName, objectKey).getObjectContent
            val inputS3String = IOUtils.toString(i, "UTF-8")
            val filePath = s"${data_lake_get_path}"
            val file = new File(filePath)
            val fileWriter = new FileWriter(file)
            val bw = new BufferedWriter(fileWriter)
            bw.write(inputS3String)
            bw.close()
            fileWriter.close()
          })
      })

额外注意事项