HDP集群中HDFS数据访问控制方案及Apache Atlas适用性咨询

Great question—this is a super common pain point when you move beyond just SQL access (like Beeline) to direct HDFS or Spark interactions. Let’s break down the solutions and best practices, plus address your Atlas question.

HDFS-Level Access Control Methods #

First, let’s fix the core issue: your Ranger Hive policies work for Beeline, but HDFS/Spark are bypassing them because they’re accessing the underlying storage directly. Here’s how to lock that down:

1. HDFS ACLs (Access Control Lists) #

HDFS’s native ACLs let you set granular permissions beyond the standard user/group/other model. For your Table_1, find its underlying HDFS path (usually /user/hive/warehouse/[db_name].db/table_1), then use ACLs to restrict access:

# Grant read-only access to a user for the view's underlying non-sensitive data (if split)
hdfs dfs -setfacl -m user:jane:r-x /user/hive/warehouse/mydb.db/v_table_1_data
# Deny access to the sensitive Table_1 path for unauthorised users
hdfs dfs -setfacl -m user:jane:--- /user/hive/warehouse/mydb.db/table_1

ACLs work for direct HDFS access and Spark, but they’re manual to manage at scale.

2. Ranger HDFS Plugin #

Since you’re already using Ranger, enable the Ranger HDFS Plugin (it’s included in HDP). This lets you create centralized Ranger policies for HDFS paths, just like you did for Hive. Here’s what to do:

• In the Ambari UI, go to Ranger > Plugins > HDFS, enable the plugin and restart HDFS services.
• Create a Ranger policy for the Table_1 HDFS path, granting access only to users who should see the sensitive data.
• Create a separate policy for the V_Table_1’s underlying data (if it’s a materialized view) or ensure the policy aligns with the view’s access rules.
  This way, any access to the HDFS path—whether via hdfs dfs commands or Spark—will be checked against Ranger’s policies.

3. HDFS Encryption Zones #

For sensitive data that needs extra protection, use HDFS Encryption Zones (EZs). EZs encrypt data at rest, and only authorized users with access to the encryption key can read the data. In HDP:

• Set up the HDP KMS (Key Management Server) if you haven’t already.
• Create an encryption key, then create an EZ for the Table_1 path:

hdfs crypto -createZone -keyName sensitive_data_key -path /user/hive/warehouse/mydb.db/table_1

Users without the key won’t even be able to read the encrypted files, regardless of ACLs.

Spark-Specific Access Control #

Spark can bypass Hive/Ranger policies if not configured correctly. Here’s how to align it with your existing setup:

• Enable the Ranger Spark Plugin via Ambari, then restart Spark services. This forces Spark (both Spark SQL and Spark Core) to check Ranger policies for data access.
• For Spark SQL, add these configs to spark-defaults.conf to integrate with Hive’s authorization:

spark.sql.hive.metastore.authorization.enabled true
spark.sql.hive.metastore.version 2.1.0  # Match your HDP 2.4 Hive version
spark.sql.authorization.enabled true

This ensures Spark SQL honors your Ranger Hive policies for tables and views.

Can Apache Atlas in HDP 2.6 Help? #

Short answer: Atlas doesn’t enforce access control directly, but it supercharges your existing Ranger setup for better data governance. Here’s how it fits in:

• Data Classification & Tagging: Use Atlas to tag sensitive columns in Table_1 (e.g., PII, Confidential). You can tag entire tables or individual columns via Atlas’s UI or APIs.
• Ranger-Atlas Integration: Create Ranger policies that reference Atlas tags instead of specific paths/tables. For example:
  • A policy that denies access to any data tagged PII for non-admin users.
  • A policy that allows read access only to data tagged Non-Sensitive.
• Data Lineage: Atlas tracks where your sensitive data flows (e.g., from Table_1 to V_Table_1 to Spark jobs), so you can audit and ensure no unauthorized pipelines are accessing sensitive data.

Atlas’s strength is in making your access control scalable and policy-driven (based on data attributes) instead of manual path/table management. But you still need Ranger to enforce the actual access restrictions.

Best Practices #

• Unify Permissions in Ranger: Use Ranger as your single source of truth for Hive, HDFS, and Spark policies—avoid mixing Ranger policies with manual HDFS ACLs.
• Leverage Views with Underlying Data Segmentation: If possible, store non-sensitive data from V_Table_1 in a separate HDFS directory with its own Ranger policy, instead of just using a logical view. This makes direct HDFS/Spark access easier to control.
• Audit Everything: Enable Ranger’s audit logs to track all access attempts (success/failure) to HDFS and Spark data. Combine this with Atlas’s lineage to get a full picture of data usage.
• Follow Least Privilege: Only grant users the minimum access they need—e.g., if a user only needs V_Table_1, don’t give them any access to the underlying Table_1 HDFS path.

内容的提问来源于stack exchange，提问作者psmith