Azure Cosmos DB Spark连接器吞吐量控制失效：未创建新容器

我尝试通过Azure Cosmos DB Spark连接器配置吞吐量控制，但运行代码后发现没有创建预期的全局控制容器。相关配置代码如下：

val changeFeedCfg = Map("spark.cosmos.accountEndpoint" -> cosmosEndpoint,"spark.cosmos.accountKey" -> cosmosMasterKey,"spark.cosmos.database" -> cosmosDatabaseName,"spark.cosmos.container" -> cosmosContainerName,"spark.cosmos.read.inferSchema.enabled" -> "false","spark.cosmos.changeFeed.startFrom" -> "Beginning","spark.cosmos.changeFeed.mode" -> "Incremental","spark.cosmos.changeFeed.itemCountPerTriggerHint" -> "100000"//optional configuration for throughput control// "spark.cosmos.throughputControl.enabled" -> "true",// "spark.cosmos.throughputControl.name" -> "SourceContainerThroughputControl",// "spark.cosmos.throughputControl.targetThroughputThreshold" -> "0.30",// "spark.cosmos.throughputControl.globalControl.database" -> "database-v4",// "spark.cosmos.throughputControl.globalControl.container" -> "ThroughputControl"
"spark.cosmos.throughputControl.useGatewayMode" -> "true")

问题根源与修复方案 #

1. 核心功能未启用 #

吞吐量控制的核心开关spark.cosmos.throughputControl.enabled被注释掉了，这直接导致功能未激活，自然不会触发容器创建逻辑。

2. 配置语法错误 #

注释块后的spark.cosmos.throughputControl.useGatewayMode配置项前缺少逗号，会导致Spark解析配置时出错，直接忽略后续配置。

3. 容器创建的前提条件 #

• 全局控制容器所在的数据库database-v4必须提前存在，连接器不会自动创建数据库
• 账号密钥需要拥有目标数据库的容器创建权限，否则无法自动生成ThroughputControl容器

修复后的完整配置代码 #

val changeFeedCfg = Map(
  "spark.cosmos.accountEndpoint" -> cosmosEndpoint,
  "spark.cosmos.accountKey" -> cosmosMasterKey,
  "spark.cosmos.database" -> cosmosDatabaseName,
  "spark.cosmos.container" -> cosmosContainerName,
  "spark.cosmos.read.inferSchema.enabled" -> "false",
  "spark.cosmos.changeFeed.startFrom" -> "Beginning",
  "spark.cosmos.changeFeed.mode" -> "Incremental",
  "spark.cosmos.changeFeed.itemCountPerTriggerHint" -> "100000",
  // 启用吞吐量控制功能
  "spark.cosmos.throughputControl.enabled" -> "true",
  "spark.cosmos.throughputControl.name" -> "SourceContainerThroughputControl",
  "spark.cosmos.throughputControl.targetThroughputThreshold" -> "0.30",
  "spark.cosmos.throughputControl.globalControl.database" -> "database-v4",
  "spark.cosmos.throughputControl.globalControl.container" -> "ThroughputControl",
  "spark.cosmos.throughputControl.useGatewayMode" -> "true"
)

额外排查步骤 #

• 检查Spark作业日志，确认是否有配置解析错误或权限不足的报错信息
• 如果不想依赖自动创建容器，可以手动在database-v4数据库下创建ThroughputControl容器，分区键建议设置为groupId（吞吐量控制功能默认使用该分区键）
• 验证账号密钥的权限范围，确保包含目标数据库的Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers/write权限

内容的提问来源于stack exchange，提问作者CHEEKATLAPRADEEP