跨双云账户的S3兼容存储：PrestoSQL需配置双Hive Catalog吗？

Short answer: Yep, you’ll need to configure two separate Hive catalogs in PrestoSQL when dealing with S3/S3-compatible buckets (like IBM Cloud Object Storage) spread across two different cloud accounts with unique HMAC key pairs.

Why this is non-negotiable (most of the time) #

A single Hive catalog in PrestoSQL uses a global set of S3 credentials defined by hive.s3.aws-access-key and hive.s3.aws-secret-key. These settings apply to every bucket you access through that catalog—there’s no built-in way to swap credentials for specific buckets within the same catalog.

This hits close to home for IBM Cloud Object Storage users, since each cloud account’s object storage instance runs with its own independent HMAC keys, and buckets are locked to their respective accounts.

Quick setup guide #

Here’s how to get this running smoothly:

1. Create two distinct catalog config files in your PrestoSQL etc/catalog folder—for example, hive-ibm-account1.properties and hive-ibm-account2.properties.
2. Populate each file with the relevant credentials and connector settings. For reference:

   # hive-ibm-account1.properties
   connector.name=hive-hadoop2
   hive.metastore.uri=thrift://your-metastore-host:9083
   hive.s3.aws-access-key=YOUR_ACCOUNT1_ACCESS_KEY
   hive.s3.aws-secret-key=YOUR_ACCOUNT1_SECRET_KEY
   hive.s3.endpoint=s3.us-south.cloud-object-storage.appdomain.cloud # Match your account's IBM COS endpoint
   

   # hive-ibm-account2.properties
   connector.name=hive-hadoop2
   hive.metastore.uri=thrift://your-metastore-host:9083 # You can reuse the same metastore if it manages tables for both accounts
   hive.s3.aws-access-key=YOUR_ACCOUNT2_ACCESS_KEY
   hive.s3.aws-secret-key=YOUR_ACCOUNT2_SECRET_KEY
   hive.s3.endpoint=s3.us-east.cloud-object-storage.appdomain.cloud # Account 2's IBM COS endpoint
   

3. Restart your PrestoSQL cluster to load the new catalogs. After that, you can query buckets from each account using syntax like ibm_account1.default.my_table and ibm_account2.default.my_other_table.

Are there any workarounds? #

While separate catalogs are the standard go-to, there are a couple of alternative paths (though they’re less straightforward for HMAC-based access):

• IAM/instance profiles: If your Presto cluster runs on cloud infrastructure that supports assuming IAM roles (like IBM Cloud VPC or AWS EC2), you could use role-based access instead of static HMAC keys. In some cases, you might be able to switch roles per query, but this requires extra setup and depends on your environment’s capabilities.
• Hadoop credential stores: You could use Hadoop’s credential store to manage multiple key pairs, but Presto’s Hive connector doesn’t natively support per-bucket credential overrides from this store. So even with this setup, you’ll likely still need separate catalogs to map to different key sets.

内容的提问来源于stack exchange，提问作者shawnzhu