能否将AWS Glue Data Catalog作为AWS托管Databricks的外部元数据存储？

Absolutely! You can absolutely use AWS Glue Data Catalog as the centralized metadata store for external services such as AWS-managed Databricks. This is a popular pattern for unifying metadata across your AWS analytics stack, letting tools like Databricks, Athena, Redshift, and EMR all share the same source of truth for table schemas, partitions, and other metadata.

Here’s how you can set this up for Databricks specifically:

• Configure your Databricks cluster to use Glue Catalog
  When launching a Databricks cluster, add these Spark configuration properties to point it to the Glue Data Catalog:

  spark.hadoop.hive.metastore.client.factory.class com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory
  spark.sql.catalog.glue_catalog com.databricks.sql.catalog.glue.GlueCatalog
  spark.sql.catalog.spark_catalog org.apache.spark.sql.delta.catalog.DeltaCatalog
  

  The first line tells Spark to use Glue’s client for metastore operations, while the other two define Glue as an available catalog in Databricks (you can even set it as the default catalog if you want to skip prefixing it every time).

• Ensure proper IAM permissions
  The IAM role attached to your Databricks cluster needs permissions to interact with the Glue Data Catalog. At minimum, it should include actions like glue:GetDatabase, glue:GetTable, and glue:GetPartition. If you need to create or update tables from Databricks, add glue:CreateTable, glue:UpdateTable, etc. For testing, you can use the managed AWSGlueConsoleFullAccess policy, but for production, always build a custom policy with least-privilege permissions.

• Use Glue Catalog tables directly in Databricks
  Once configured, you can query Glue Catalog tables just like you would with Databricks’ native metastore. For example:

  -- Query a table from the Glue Catalog
  SELECT * FROM glue_catalog.my_database.my_s3_table;
  
  -- If you set Glue as the default catalog, skip the prefix
  SELECT * FROM my_database.my_s3_table;
  

A few extra tips to keep in mind:

• Version compatibility: Most modern Databricks Runtime versions support Glue Catalog integration, but double-check the docs for your specific runtime to avoid compatibility gaps.
• Cross-account access: If you need to access a Glue Catalog in another AWS account, set up cross-account IAM roles and adjust the Spark config to specify the target account ID.
• Unified workflow: Any changes to tables (like adding partitions or updating schemas) made via Glue, Athena, or Databricks will sync across all tools using the catalog, keeping your metadata consistent.

内容的提问来源于stack exchange，提问作者Obaid