启用EMR本地磁盘加密：开源HDFS加密配置及CloudFormation部署方法

Got it, let's break this down into two clear, actionable parts: first the open-source HDFS at-rest encryption setup, then how to embed that configuration into your EMR CloudFormation template.

This is vanilla Hadoop configuration, so it works across any distribution (including EMR's open-source base). Here's the step-by-step:

Prerequisites #

• Ensure your Hadoop version is 3.x or newer (older versions have limited encryption support)
• Have a Key Management Service (KMS) running—you can use Hadoop's built-in open-source KMS or a compatible third-party KMS. We'll focus on the built-in one here.

Step 1: Configure Core-Site.xml #

First, point Hadoop to your KMS instance:

<property>
  <name>hadoop.security.key.provider.path</name>
  <value>kms://http@<your-kms-host>:16000/kms</value>
</property>
<property>
  <name>hadoop.security.crypto.cipher.suite</name>
  <value>AES/CTR/NoPadding</value> <!-- Default recommended cipher -->
</property>

Step 2: Configure HDFS-Site.xml #

Enable encryption zones and set KMS integration for HDFS:

<property>
  <name>dfs.encryption.key.provider.uri</name>
  <value>kms://http@<your-kms-host>:16000/kms</value>
</property>
<property>
  <name>dfs.encryption.zones.enabled</name>
  <value>true</value>
</property>
<property>
  <name>dfs.encrypt.data.transfer</name>
  <value>false</value> <!-- Set to true only if you need in-transit encryption too -->
</property>

Step 3: Create Encryption Keys and Zones #

Once the cluster is up, run these commands on the master node:

1. Create a new encryption key (stored in KMS):

   hadoop key create my-hdfs-encryption-key
   

2. Create an encryption zone (EZ) tied to this key—any data written here will be encrypted at rest:

   hdfs crypto -createZone -keyName my-hdfs-encryption-key -path /user/encrypted-data
   

Step 4: Verify Encryption #

Check if the zone is active:

hdfs crypto -listZones

Upload a test file and confirm it's encrypted:

hdfs dfs -put test.txt /user/encrypted-data
hdfs dfs -ls -e /user/encrypted-data

You'll see an encrypted tag next to the file if setup worked.

EMR lets you inject custom configs directly via the Configurations field in your CloudFormation template—no need to manually edit files on nodes. Here's how to do it:

CloudFormation YAML Snippet #

Add this to your AWS::EMR::Cluster resource:

Resources:
  MyEncryptedEMRCluster:
    Type: AWS::EMR::Cluster
    Properties:
      Name: "EMR-With-HDFS-Encryption"
      ReleaseLabel: emr-6.15.0 # Use a compatible EMR release (matches Hadoop 3.x)
      ServiceRole: !Ref EMRServiceRole
      JobFlowRole: !Ref EMRInstanceProfile
      InstanceGroups:
        # Your instance group configs (master, core, task nodes)
      Configurations:
        # Configure HDFS-site for encryption
        - Classification: hdfs-site
          Properties:
            dfs.encryption.key.provider.uri: "kms://http@<your-kms-host>:16000/kms"
            dfs.encryption.zones.enabled: "true"
            dfs.encrypt.data.transfer: "false"
        # Optional: Configure core-site for KMS integration
        - Classification: core-site
          Properties:
            hadoop.security.key.provider.path: "kms://http@<your-kms-host>:16000/kms"
            hadoop.security.crypto.cipher.suite: "AES/CTR/NoPadding"
      # Optional: Add a step to auto-create encryption keys/zones on cluster launch
      Steps:
        - Name: "Create-HDFS-Encryption-Zone"
          ActionOnFailure: CONTINUE
          HadoopJarStep:
            Jar: "command-runner.jar"
            Args:
              - bash
              - -c
              - |
                hadoop key create my-hdfs-encryption-key || true
                hdfs crypto -createZone -keyName my-hdfs-encryption-key -path /user/encrypted-data || true

Key Notes #

• Replace <your-kms-host> with the private IP of your KMS node (or a DNS name if you've set one up)
• If you're running the Hadoop KMS on the EMR master node, use localhost as the host
• The Steps block ensures your encryption key and zone are created automatically when the cluster starts—no manual intervention needed
• Make sure your EMR instance roles have permissions to access the KMS (if using a remote KMS)

内容的提问来源于stack exchange，提问作者fledgling