如何通过Knox实现节点认证以访问Hadoop服务（非LDAP方式）

Awesome question! Skipping LDAP username-password auth and going with node-level authentication via Knox makes total sense for scenarios where you trust entire nodes rather than individual users. The most secure and straightforward approach here is SSL client certificate authentication—here's how to set it up step by step:

Step 1: Generate CA and Node Client Certificates #

First, you'll need a Certificate Authority (CA) to issue trusted certificates for your nodes. You can create a self-signed CA for internal use, or use an enterprise CA if you have one.

• Create a CA keystore:

  keytool -genkeypair -alias knox-ca -keyalg RSA -keysize 2048 -validity 3650 -keystore knox-ca.jks
  

• Export the CA certificate to share with Knox and nodes:

  keytool -exportcert -alias knox-ca -file knox-ca.crt -keystore knox-ca.jks
  

• Generate a client keystore for each node (replace node1 with your node identifier):

  keytool -genkeypair -alias node1-client -keyalg RSA -keysize 2048 -validity 3650 -keystore node1-client.jks -dname "CN=node1.example.com,OU=Engineering,O=YourOrg,L=City,ST=State,C=US"
  

• Generate a Certificate Signing Request (CSR) for the node:

  keytool -certreq -alias node1-client -file node1-client.csr -keystore node1-client.jks
  

• Sign the CSR with your CA:

  keytool -gencert -alias knox-ca -infile node1-client.csr -outfile node1-client.crt -keystore knox-ca.jks -ext san=dns:node1.example.com,ip:192.168.1.10
  

• Import the CA certificate and signed node certificate into the node's keystore:

  keytool -importcert -alias knox-ca -file knox-ca.crt -keystore node1-client.jks
  keytool -importcert -alias node1-client -file node1-client.crt -keystore node1-client.jks
  

Step 2: Configure Knox Gateway to Require Client Certificates #

Update Knox's core configuration to enforce client certificate authentication and trust your CA:

1. Edit $KNOX_HOME/conf/gateway-site.xml and add/update these properties:

   <property>
       <name>gateway.ssl.enabled</name>
       <value>true</value>
   </property>
   <property>
       <name>gateway.ssl.need.client.auth</name>
       <value>true</value>
   </property>
   <property>
       <name>gateway.ssl.truststore.type</name>
       <value>JKS</value>
   </property>
   <property>
       <name>gateway.ssl.truststore.path</name>
       <value>/path/to/knox-ca.jks</value>
   </property>
   <property>
       <name>gateway.ssl.truststore.password</name>
       <value>your-truststore-password</value>
   </property>
   

2. Restart the Knox gateway to apply changes.

Step 3: Map Certificates to Node Identities in Knox #

Knox needs to extract a valid identity from the client certificate to pass to Hadoop services. Configure the ClientCertAuthenticationProvider in your topology:

Edit $KNOX_HOME/conf/topologies/your-topology.xml (replace your-topology with your actual topology name) and add this authentication provider:

<provider>
    <role>authentication</role>
    <name>ClientCertAuthenticationProvider</name>
    <enabled>true</enabled>
    <param>
        <name>principal.mapping.pattern</name>
        <value>CN=(.*?),OU=.*</value>
    </param>
    <param>
        <name>principal.mapping.value</name>
        <value>$1</value>
    </param>
</provider>

The principal.mapping.pattern uses a regex to extract the node's identity from the certificate's Distinguished Name (DN). Adjust the regex to match your certificate's DN structure (e.g., if your CN is node1, this pattern will extract node1 as the identity).

Step 4: Configure Hadoop Services to Trust Knox's Proxy #

Knox acts as a proxy between your node and Hadoop services, so you need to allow Knox to proxy the node's identity:

For example, in HDFS (hdfs-site.xml):

<property>
    <name>hadoop.proxyuser.knox.hosts</name>
    <value>knox-gateway.example.com</value> <!-- Restrict to your Knox host -->
</property>
<property>
    <name>hadoop.proxyuser.knox.groups</name>
    <value>*</value> <!-- Or restrict to specific node groups -->
</property>
<property>
    <name>hadoop.proxyuser.knox.users</name>
    <value>node1.example.com,node2.example.com</value> <!-- List your trusted nodes -->
</property>

Repeat similar configurations for YARN, MapReduce, or other Hadoop services you're accessing via Knox.

Step 5: Test the Node Authentication #

From your node, send a request to Knox using the client certificate. For example, using curl:

# First, export the certificate and key from the node's keystore to PEM format
keytool -importkeystore -srckeystore node1-client.jks -destkeystore node1-client.p12 -srcstoretype JKS -deststoretype PKCS12
openssl pkcs12 -in node1-client.p12 -nodes -nocerts -out node1-client.key
openssl pkcs12 -in node1-client.p12 -nodes -nokeys -out node1-client.crt

# Send a test request to WebHDFS via Knox
curl --cert node1-client.crt --key node1-client.key --cacert knox-ca.crt https://knox-gateway.example.com:8443/gateway/your-topology/webhdfs/v1/?op=GETHOMEDIRECTORY

If everything is configured correctly, you'll get a valid JSON response from HDFS without needing any username/password.

内容的提问来源于stack exchange，提问作者Abhishek Tiwari