非Kerber化Hortonworks集群访问Kerber化Cloudera集群的可行方案问询

Great question! Let’s break this down step by step based on my hands-on experience integrating Hortonworks (HDP) and Cloudera (CDH) clusters.

Here are the most practical approaches depending on your use case:

• Leverage CDH's gateway services with non-Kerberos auth
  CDH services like HDFS and YARN can be configured to expose gateways (e.g., HttpFS for HDFS) that allow simple authentication—think username/password or IP whitelisting. This lets your HDP cluster’s clients access CDH resources via HTTP interfaces without full Kerberos setup. It’s great for lightweight, low-security needs, but make sure to restrict access to trusted HDP nodes only.

• Configure Kerberos clients on HDP nodes for temporary access
  You don’t need to Kerberize the entire HDP cluster. Just install Kerberos client packages (krb5-workstation, krb5-libs) on HDP nodes that need access, then point their krb5.conf to CDH’s KDC. Users or scripts can run kinit with a valid CDH Kerberos principal to get a ticket, then interact with CDH services normally (e.g., hdfs dfs -ls hdfs://cdh-nn:8020/). Perfect for one-off data transfers or ad-hoc queries.

• Use Apache Knox as a secure proxy
  Knox acts as a unified gateway that handles authentication translation. Deploy Knox between the two clusters: configure it to authenticate against CDH’s Kerberos realm, while exposing non-Kerberos auth methods (like Basic Auth) to HDP clients. HDP traffic goes through Knox, which handles fetching Kerberos tickets and forwarding requests to CDH. This is the most secure option for long-term cross-cluster access, as it centralizes access control.

• DistCp with Kerberos parameters for data sync
  For bulk data migration, DistCp supports Kerberos-aware transfers. Set up krb5.conf on HDP nodes to point to CDH’s KDC, run kinit with a CDH principal that has access to both clusters’ HDFS, then execute DistCp with Kerberos-specific flags. Example command:

  kinit hdp-sync-user@CDH.REALM
  hadoop distcp hdfs://hdp-nn:8020/source-dir hdfs://cdh-nn:8020/dest-dir -Ddfs.namenode.kerberos.principal=hdfs/cdh-nn@CDH.REALM
  

Absolutely—this is a common approach for integrating clusters long-term. Here’s what you need to do:

• Ensure all HDP nodes can reach CDH’s KDC over network ports 88 (UDP/TCP).
• Update krb5.conf on every HDP node to use CDH’s Kerberos realm as either the default or a trusted realm. If the clusters have different domain names, add realm-to-domain mappings in krb5.conf.
• Work with your CDH KDC administrator to create Kerberos principals (SPNs for HDP services, UPNs for HDP users) in CDH’s realm. For example, create hdfs/hdp-nn@CDH.REALM for HDP’s NameNode, or john.doe@CDH.REALM for an HDP user.
• This eliminates the need to manage two separate KDCs, streamlining authentication management across both clusters.

Yes—this is the most robust long-term solution if you want to fully secure the HDP cluster and enable seamless integration with CDH. Here’s the high-level process:

1. Choose a KDC: You can use CDH’s existing KDC (as the universal KDC above), deploy a new standalone KDC, or set up cross-realm trust between CDH’s KDC and a new HDP KDC.
2. Install Kerberos clients on HDP: Every HDP node needs Kerberos client packages, with krb5.conf configured to point to your chosen KDC.
3. Create SPNs and UPNs:
   • For each HDP service (NameNode, ResourceManager, HBase Master, etc.), create corresponding SPNs in the KDC (e.g., yarn/hdp-rm@REALM).
   • Create user principals (UPNs) for HDP users, or sync them from an existing LDAP directory if you have one.
4. Enable Kerberos via Ambari: Use HDP’s Ambari management console to run the Kerberos setup wizard. It will auto-configure all HDP services with Kerberos settings, generate keytab files, and restart services.
5. Validate: After setup, test authentication with kinit and verify access to HDP services, then confirm cross-cluster access to CDH works with the shared KDC or trust relationship.

• Note: Plan this during a maintenance window, as services will need to restart. If using CDH’s KDC, coordinate with the CDH admin to create the required principals and keytabs.

内容的提问来源于stack exchange，提问作者developerqueries