在Databricks同一集群策略中兼容Serverless与SingleNode场景：Terraform中能否按需启用Process Isolation配置？

Great news—you don’t need separate policies or manual edits! Databricks cluster policies support conditional logic based on the cluster type, which lets you apply the processIsolation config only to Serverless clusters while reusing the same policy for SingleNode clusters.

How it works #

Databricks exposes the ${clusterType} variable in cluster policies, which resolves to:

• serverless for Serverless clusters
• single_node for SingleNode clusters

You can use this variable with a condition field in your policy definition to toggle the processIsolation configuration on or off based on the selected cluster type.

Terraform configuration example #

Here’s a complete Terraform resource that implements this logic:

resource "databricks_cluster_policy" "unified_serverless_singlenode" {
  name = "Unified Serverless + SingleNode Policy"
  definition = jsonencode({
    # Core: Apply processIsolation ONLY to Serverless clusters
    "spark_conf.spark.databricks.processIsolation.enabled" = {
      "type": "boolean",
      "value": true,
      "hidden": true,
      "condition": "${clusterType == \"serverless\"}"
    },

    # Enforce cluster type selection (optional but recommended)
    "spark.databricks.cluster.profile" = {
      "type": "string",
      "allowedValues": ["serverless", "single_node"],
      "defaultValue": "single_node"
    },

    # Shared configuration for both cluster types
    "autotermination_minutes" = {
      "type": "int",
      "value": 60,
      "defaultValue": 60
    },

    # SingleNode-specific: Force 0 workers (since it's a single node)
    "num_workers" = {
      "type": "int",
      "value": 0,
      "hidden": true,
      "condition": "${clusterType == \"single_node\"}"
    }
  })
}

Key details to note: #

• The condition field on the processIsolation config ensures it’s only applied when the cluster type is serverless. For SingleNode clusters, this config won’t appear in the cluster’s spark settings at all.
• We’ve added an allowedValues restriction on spark.databricks.cluster.profile to limit users to only the two cluster types you care about—this prevents unexpected cluster types from using the policy.
• The num_workers config is conditionally set to 0 only for SingleNode clusters, which aligns with the SingleNode profile requirements.

Testing the policy #

After applying this Terraform configuration:

1. Create a new cluster using the policy and select "Serverless" as the cluster type. Check the Spark configs—you’ll see spark.databricks.processIsolation.enabled=true applied.
2. Create another cluster and select "SingleNode"—the processIsolation config will be missing, and the cluster will use the shared settings like autotermination.

This approach keeps your policy unified, avoids manual edits, and scales cleanly if you need to add more conditional logic later.

内容的提问来源于stack exchange，提问作者Arthur Clerc-Gherardi